Cerberus is full of BS, I'll give them that, But they are fun to fight and most of all, fun to kill, with with all their screaming and moaning and everything. I've never found phantoms to be too annoying, not even on host. If anything, because lag is usually bad for me, I find them more annoying off host, since you can't predict what's going to happen next. The only thing that I really hate about phantoms is that they are pretty much the only mook(and I use the term lightly, since they are more of a boss unit than Atlas) enemy that's totally immune to Overload.
If I had to choose most annoying Cerberus unit, it would have to be Nemesis. Pretty much impossible to hit them with projectile power, their shields sometimes seem tougher than those of Centurions, the way they are able to zero on you while they are running, they are there more to annoy you than threaten you... yeah, that's basically it. They are more annoyance then anything else.
***
Reapers are surprisingly fair... the fairest of all enemy factions, which is weird, given they are supposed to be the main antagonists and how they supposedly balk at nothing. Other than an occasional magnet hand from a banshee and the BS with Marauder elbows, yeah, I generally have no problems with Reapers.
My most hated enemy from Reapers is Banshee. Their screams were terrifying at first. After almost 700 hours of MP, they are just annoying and double banshee waves on gold are a real pain in the ass.
***
Geth... **** that **** to the nether pits of the hell that spawned them. From hunters timing their shots to cancel your reloading or power usage and them sometimes being totally immune to pretty much this side of Cobra, to rocket cheaters firing three missiles the second they spawn and bombers ignoring Overloads, Pyros flame broiling you from a mile away and primes sniping you from across the map, blind sighting you with turrets and oh, more stagger, big surprise there with those demonic drones...
Which geth unit I don't hate? I guess troopers are kinda tolerable.
***
Collectors... what's not to love? Those pesky captains totally ignoring your shield and health gate(I really love the idea of them having Marksman as passive, fits nicely)? Scions carpet bombing you... from across the map... around the corner? Flying ninjas that are able to sync kill(with no reliable trigger to read) you without giving you the slightest idea of them being there and firing butt laz0rs at you(after they die even) that totally ignore cover sometimes? Their demonic pets that take your powers away from you? The fact that, just as you're about to kill them, they get possessed and make you do half as much work again? I guess abombs are kinda fun, but I've never seen an appeal of it. And if everything else fails, they'll pull the victory out of your grasp by killing your console.
I actually find Collectors easy. That is, up to one point, where the damn bugs decide that enough is enough and do some sort of trick that will make your team look like a lobotomy has been performed on them and suddenly everybody will die.
There's not a single collector unit I don't hate.