The question then would be why did the Leviathan didn't go back to Andromeda when they were hunted by the Reapers. Also, why didn't the Reapers know about the mass relay, if they were built to manage all the affairs for the Leviathan (overlooking and regulating the servant species).
I'm not 100% sure but I remember it being said ingame that the Reapers built the Mass Relays, which means they didn't exist prior to their harvest of the Leviathans. Of course, the Reapers lie (or their statements are turned into lies retroactively because the writers have no consistency), since Sovereign stated they had no creator.
The Reapers built the Citadel, and the relay network, yes. But that doesn't necessarily mean they invented the tech. I mean, they created indoctrination as a copy of the Leviathan ability to control organics, after all. Maybe the Leviathans had relay tech, but the Reapers expanded on it.
But like I said, there are wrinkles to iron out. Maybe all data of the relay was lost (like what happened to Ilos) maybe it was thought to have been destroyed? Perhaps some Leviathans did escape before knowledge of the relay was lost. And their descendants are the 'Remnants"?