Tl;Dr it's science fiction. Universal translators are a literary convention that doesn't make much sense, but still makes much more sense than alternatives, and is canon until specifically stated otherwise.
Canon does state otherwise. We find out that there is no physical language barrier with the exception of the Hanar, because that's what makes them unique. If it wasn't unique, then that wouldn't be their distinguishing trait. That's why there's a trade language. It's something every species (other than the non-vocal Hanar) can communicate in. Even if it is a bunch of grunts and clicks. Translator tech can still probably be used in a lot of contexts, but there's nothing in-game that demonstrates it's what most people (besides the Hanar) use.
Not once in the lore regarding various languages does it say that other species can't communicate in such a language, with one exception: the Hanar, who I'll cede here. Being perfectly honest, I forgot about them because they're so peripheral to the rest of the story, but there's certain things that translators don't account for, like all Elcor, who announce their emotions before they speak, and the "fly in the lotion" Turian on Noveria. If translators were used consistently, it'd be able to recognize the Elcor and would be able to deal with relatively easy language differences.
If translator technologies were being used consistently, we wouldn't have accents, malapropisms, and idioms being spoken through the in-game language. It's also the reason why Javik is able to understand everyone when he touches Shepard. If we were getting translators on everyone, there's no way Javik could understand all of them. But he can, because they're speaking the same language.
Long story short, translators are a means to handwave contact between isolated populations, but most people in cosmopolitan space i.e. the Citadel and the Normandy, are communicating to one another directly.