I dunno if that's viable. A race that loses 99% of its population is never going to get back to where it would have been unless someone's carved out a preserve for it. If you lose an early war in a 4x game, it's typically game over. (I pulled it off once in a MoO 1 game, but it's an exception that proves the rule; the Sakkras and Klackons went all-out with bioweapons and depopulated half the Galaxy, so in effect I had a preserve to develop into.)
The Quarians losing 99% of their population in a single battle, would be a bit silly though. There are multiple examples in the games of fleets retreating through Mass Effect relays after a defeat, including against the Reapers, so the Quarians not being able to do that against the Geth seems rather contrived.
Not to mention that it would be completely mad for the Quarians to commit every ship they have, including the civilian ships, to a battle they may lose. Of course ME3 says the Quarian Admirals actually are that stupid, but I wouldn't mind that getting a retcon. The Quarians putting all their eggs in one basket was forced just to give Shepard a choice between destroying the Geth or allowing the Quarians to go extinct.
I'd just have Shepard's choices determine whether Rannoch is owned by the Quaians, the Geth, or shared, and whether the Quarians are still wandering vagabonds or have recolonized Quarian space.