There are some things to keep in mind with Agents.
1. They reduce the amount of time the specialist they are placed under takes to complete missions.
2. Some unlock war table operations.
3. Some will unlock shops that you cannot obtain without them (Master Dennet as as example gives you a mount store once recruited.)
4. They count as a perk in whatever specialist tree they are in, reducing the number of perks you need to spend in that tree to unlock the higher level perks. (having 4 agents that are in Cullen's tree will allow you to grab the advanced focus perk without having ever actually spent any other points in that category previously.)
Do they affect story:
Not really. They are optional. Because of this, some of them become unavailable to be unlocked after certain points of the game.