The mods are limited to retextures of the pyjamas model though, they don't let you wear your armour/adventuring gear.
That was exactly my point. The retextures can make them look like just about anything. I'm particularly fond of one of the Qunari outfits that someone made:
CE limits you to your adventuring outfit. I used to use it specifically for that, just like the OP. Then I saw a picture of one of the mods and thought that it was a great idea. The Inquisitor looks terrible in pajamas, but wearing your adventuring/combat gear when you're just casually strolling around your fortress doesn't make a lot of sense either. I like that the various mods give you a nice set of options for clothing appropriate to the Inquisitor's duties within Skyhold's walls.
I was putting it out there in case the OP was unaware of the option.