I'm not sure if the ME universe is doomed to not be a giant, long-lasting franchise, but if it is I don't think it's the ending of ME3 that doomed it. I think it was the use of the Reapers as the central conceit of the series.
The Mass Effect universe is big, sure, but big's easy to do in fiction. If anything, it's positively triveal. You can make a series set about the various stories of New York City a larger and and more diverse setting than ME has ever been written out as. More important than size of the setting, I suspect, is the underpining idea that drives it: what makes the setting of sci-fi Series A distinct from the setting of sci-fi Series B?
Some series go with the setting itself: Fallout is post-apocolypic retro-sci-fi America. Dragon Age is about the continent of Thedas in the, well, Dragon Age. Star Trek is about space as the Final Frontier, with a heavy emphasis on the nigh-infinite worlds and constant exploration into the unknown.
Mass Effect, though? Mass Effect rested on the threat of the Reapers as the unifying framing device. Once they were resolved, the series was always going to have to fall back on another unifying idea for what makes Mass Effect, well, Mass Effect... and I don't think the writers ever decided what that was. There was too much tone shifting and focus-jumping across the series.
ME1 was a rather ideological and political thriller: it might not warrant being called a political thriller, but the narrative of a rising nation-state and international political relations was central across the story: the relations between species was as important as the relations between characters, and the themes of the narrative were sharply kept in the Paragon/Renegade ideologies.
ME2 cast that aside for it's aimless romp in the Terminus. Cerberus became a glaring example of a species-centric ideology, but at the cost of eclipsing the diversity of the Alliance and lacking any real other-species contenders. The Terminus was, if anything, marginalized by the very game supposedly focusing on it, and the sole Terminus polity we were introduced to was never justified as to why we should consider it important. Relations between species were thrown aside to focus on the cast and their personal relations with Sheaprd, and the former ideologies of ME1 were thrown aside for a tone-based morality system that jumped positions frequently.
ME3 moved away from ME2 and back towards ME1 in a lot of ways, but only somewhat. Inter-species relations played a part for a little while, but it was mostly isolated in two specific subplots. Political tensions were replaced by the grand strategy of the war. And while the morality system did become less done-driven, it also still lacked the ideological component of ME1.
So, really, when we get past the superficial differences in the aliens, what is it that distinguishes Mass Effect from other sci-fi? I don't think it's the Mass Effect: mass effect drives and relays, while setting-specific, are primarily just tools to justify the interaction between species: little more than hyperdrives and anti-gravity machines common across sci-fi.
To me, it's the Reapers: the looming apocolyptic threat. That's what unifies the franchise, and that's what every previous work more or less worked within. Once it was gone, and it was always going to be gone by the end of the trilogy, the ME universe was either going to have to find a way to bring back that common denominator (possibly by introducing a similar underlying apocolyptic force), find a new one, or try and do without a unique hook. It's the last one that I would think truly 'dooms' a franchise.
So, to go back to the start, no. I don't think the ending doomed the ME franchise. It could even well continue on a variety of paths: it could try to be a lot of ME2-style companion-based games. It could be a standard sci-fi space opera with political focuses. It could do something else.
But I think the biggest anchor around its neck from this time forward comes from the loss of the Reapers as the Big Bad. Not how they ended.