I did a FF play through on nightmare difficulty about a year ago for fun. I absolutely hated it. Here's the semi-end game build I used for Varric anyway.
Artificer Tree: 4 Points
- Spike Trap (No Upgrade)
- Opportunity Knocks
- Fallback Plan (No Upgrade)
- Tricks of the Trade
Double Dagger Tree: 9 Points
- Flank Attack (Skirmisher Upgrade)
- Twin Fangs (Ripping Fangs Upgrade)
- Bloodied Prey
- Dance of Death
- Sneak Attack
- Deathblow (Thrill of Victory Upgrade)
Sabotage Tree: 3 Points
- Poisoned Weapons (Infected Wounds Upgrade)
- Fighting Dirty
Subterfuge Tree: 7 Points
- Steath (No Upgrade)
- Easy to Miss
- Evasion
- Evade (No Upgrade)
- Ambush
- Shadow Strike (Quick Blade Upgrade)
Total Points: 23 (Beyond that, you can do whatever you want, though I'd recommended getting "Looked Like it Hurt" + "Cheap Shot" from the Sabotage tree.)
Skill slots are as follows: Fallback Plan, Flank Attack (Favourited), Twin Fangs, Deathblow, Poisoned Weapons (Favourited), Evade (Favourited), Shadow Strike, and the last slot can either be filled with Stealth or whatever else you might want.
I find that dagger rogues are pretty self sufficient on the battlefield, and require minimal micro-managing with Flank Attack/Evade favourited since they'll pretty much always be in stealth, especially Varric.
I'd also suggest using Fade-Touched Bloodstone (7.5% Damage), Fade-Touched Summer Stone (5% Damage), and Fade-Touched Dawnstone (Heal 25% Damage Taken) as your masterworks. I know that most people might be more inclined to use the Hidden Blades MW(s) instead of the damage ones with rogues, but given that this build focuses more on landing a few heavy hits rather than multiple weaker ones, FTB and FTSS will end up being more beneficial than the rarely activating Hidden Blades. FTD should speak for itself.
I can recommend a gear setup too, if you want.
Edit: As a side note, I'd also highly advise using Cole or Sera instead of Varric (unless you really want to use him for his personality and attitude). FF puts a serious damper on Varric's DPS, and makes it difficult for him/artificers to really shine.