Varric can't leave. You need to be able to have a full party.
A lot of people got confused with this because of the way a trailer was edited. The narrator said companions can leave, whilst showing a Varric walking away from the The Inquisitor. Those that believe this is the scene where he leaves seem to forget the scene immediately after taking Skyhold and Varric telling you about his "friend."
You never actually need a full party. The only time you actually need a class other than your own is when trying to access optional content, like lighting veilfire torches if you're not a mage, or unlocking doors for rogues and kicking through walls for warriors. None of these things are ever essential to complete the main story, however. The only time Varric is ever essential to the story is when he contacts Hawke, but after that, his purpose is fulfilled and the story can go on without him.
For Dorian's quest, I went only with him and no others. Seemed to make sense that way. For Sera's Verchiel quest, I only took her and it was just the two of them fighting through the ambush. I thought it was more fun this way.
For the final battle against Corypheus, I actually solo'd this with my Knight-Enchanter, because I liked the idea that Corypheus scooped her up and separated the entire group, forcing her to fight against the monster on her own. It made her descent after her victory seem more heroic, not to mention it better explains why your companions are suddenly nowhere to be found after the orb is destroyed.