I don't see how that adds up. Making them optional doesn't minimize the number of companions you can recruit. Having companions as optional simply allows the player to determine whether or not it makes sense for that respective protagonist to recruit the person. Maybe a pro-templar protagonist might not recruit apostates who are vying to help the mages maintain their autonomy from the Chantry and the templars. Perhaps a Dalish Inquisitor wouldn't want to recruit a chevalier who murdered elves in cold blood. It's about what makes sense for the character.
You seem to be talking strictly from an roleplaying perspective, which is fine. I'm more worried about the practical side of things. The fact is, by opting to skip a companion, you are implicitly telling the game you'd prefer to forgo any and all content related to that character, owing to your personal vision of the character you're playing. If that's really something you would like, then I can't convince you otherwise. But come to think of it, why would these characters even be companions if it didn't 'make sense' for the Inquisitor, no matter their beliefs or affiliations, to recruit them in the first place?
Don't get me wrong, I'm definitely with you that player agency is a wonderful thing to have. However, in some cases, I think it can backfire in subtle ways.
I've tried skipping companions in DA:O and ME2. I would be lying if I said I enjoyed those playthrough's as much as I did those where I recruited everyone.