The lack of an in-your-face kind of enemy may have been intentional. Shepard's war is a shadow war after all.
The discussion with
Vigil on Illos mentioned a tactic of the Reapers was to send indoctrinated individuals of the target spieces to infiltrate the target before the attack comes. It makes me think of Horizon when they lost communications. The stealth of a predator has been the Collector's method of attack The Collectors don't want you thinking about them constantly. Makes their next strike easier.
Collectors like the Stealth, the Bobbie Traps, the Trojan Horses and the Slight of Hand type tactics.
The Collector threat was mostly to soft targets like undefended colonies. With the swarms, a single collector cruiser was sufficient for taking all colonials. They were not limited to soft targets though. Small hard targets like Normandy (SR1) could be handled early in ME2 by the Collectors. The Collector tech, which is actually based on Reaper tech, made them look much more capable as they were successfully striking targets accross the Terminus Systems.
Note an observation: Tthe space station may have been just a re-purposed Collector Dreadnaught instead of an actual space station. I say that because of the hologram Shepard and his team had up in the conference room as they planned their attack on the base. That hologram depicts the same sillouette as the Collector Cruiser as seen standing on it's nose.
The Collectors themselves were too few in number to be a major threat by themselves. After all they seemed to be the crews of just one cruiser and a space station. To augment their numbers, the collectors hired local mercenaries. You could buy off a merc, and the credits certainly did not mean anything to the Collectors.
1.) The most obvious case of Collectors using Mercs happened during Mordin's recruitment mission. The Blood Pack on Omega was trying to destroy the Blue Suns and the non-human elements on Omega. The Vorcha in particular seemed interested in taking Omega for their own so indoctrination was not required.
2.) The Blue Suns never spoke up on the topic of working for the Collectors.
A.) However, Zaeed trying to kill Santiago may have kept Santiago below the Collector's radars so one stop shopping for control was never set up with the Blue Suns.
B.) Elements of the Blue Suns fought each other for control of various operations, and many of these elements seemed very interested in Prothean Technology. Why, other than maybe credits, is unclear.
C.) There was also the elements of the Blue Suns that seemed intent on accessing and gaining control of the Citadel via Harkin. These Blue Suns may or may not have been trouble waiting to happen if they had techs there to fix the gate to Dark Space.
D.) Oh, yes; lest we forget the Warden's attempt on the prison barge Purgatory to capture Shepard for sale to his customer. It may have been the Shadow Broker or perhaps the Collectors directly. Not sure really not that it made a difference to the end destination.
3.) During the Archangel mission, Shepard can recover an Eclipse datapad. This datapad indicated all three merc groups, Blood Pack, Blue Suns, and Eclipse were preparing to take Aria's organization down. Hopefully giving Aria the datapad allowed her to clean that mess up before the Collector's could make the merc occupied Omega a forward base and listening post. In other missions, elements of Eclipse may nave been sent specifically after Cerberus. For example, the mission in which Shepard is sent to recover the captured Cerberus Operative.
A parting thought - When you add the use of the mercs, I think the Collectors are an on-going threat.