On the back of flanker Hamish Watson being sidelined, this news comes as another blow following their opening 27-3 pool defeat to Ireland.
The 27-year-old will be replaced in Scotland's squad by experienced Edinburgh scrum-half Henry Pyrgos, who won the last of his 27 caps in November 2017.
He joins Greig Laidlaw and George Horne as Gregor Townsend's half-back options for their remaining games at the tournament.
"We are disappointed for Ali to have to return home so early in the tournament," said head coach Townsend, whose side face Samoa next up.
"Ali and Hamish had invested a lot of effort into being in their best physical shape for the World Cup and it's a shame that they've only been involved in one game.
"However we have a lot of belief in our wider group and the two new players (Magnus Bradbury and Pyrgos) who have been given this opportunity."
After Monday's match against Samoa in Kobe City, Scotland face Russia on Wednesday, October 9 and then hosts Japan on Sunday, October 13.