Oh, I got all these agents. I think I have 12 or 13 all together at the moment.
So far I have been doing every single quest. I cleared all but 2 maps, killed 4 or 5 dragons, did all the companion quests and the only remaining main quest is the assassination of Empress Celene. You know, the really, really urgent mission I totally should do before it's too late. 
A good example for a side quest that doesn't matter would be bringing flowers to the grave of the old elf's wife. I also don't see how this translates into more power for the Inquisition... but okay.
I enjoyed Skyrim a lot although I only completed it once. Every single quest done. It's an entirely different concept than DA:I, one that starts with a truly open world and doesn't just pretend to have one to check another item on the feature list.
A little quest like that doesn't really matter in the grand scheme of things (though I imagine things like that would gain some goodwill with the people, hence power and/or influence) but it might be something that's important to your inquisitor personally. Plus, it's flavor. Detail. Something small and personal to remind you of what the "little people" are going through in this war. Here's a poor man who can't bring flowers to his wife's grave anymore because of all the mess. IIRC, isn't it your Inquisitor who offers to do it? I don't think he actually asks you.
As for the empress, well, the ball is at a set date, right? Essentially you're time skipping ahead, or not, it depends on when you imagine that date to be. It could very well be months away, leaving you lots of time to do other things.
The reason (well, one of them) I stopped playing Bethesda games was because of Oblivion. In Patrick Stewart's dying breath he gives you this terribly important, world-saving quest, which you can then proceed to completely ignore unless/until you actually ever feel like getting to it. I have no idea if Skyrim handled whatever world-saving mission your PC is on any better, but seriously.
The buffet analogy is good, but you're still going to have completionists who need eat everything on the buffet. From an RP perspective it gives you a lot of options to get the power and influence you need to get to the next major story point. From a purely gameplay perspective, I feel overwhelmed by the sheer amount of stuff that's in my journal. I'm level 16 and I there are a couple of things I don't even know if I should bother doing them at this point. But there they are, sitting in my journal and glaring at me, all unfinished.
For example, I'm in Emprise du Lion right now. It's fine and it's a good story mission, but why do I need to be there? The only reason I'm there is because it's a place on the map. I have more than enough power to get to the next storyline mission, but it feels like I'm just killing time.
But isn't the reason for going to Emprise du Lion that there's signs of a big Red Templar operation there? Seems not unrelated to the main goal of the Inquisition, even if it's not essential to the main quest. That's what I remember, anyway.
I guess the difference is that I'm not really a completionist. So I don't mind quests staying open in the quest log, and I have no trouble not finishing one if it doesn't fit my Inquisitor's personality. In fact, I barely check the quest lot at all. I usually check the map for any quests in the area that I might be interested in.