That's just the thing - I want very different casual outfits for each of my PCs, ranging from full armour for the more aggressive/paranoid ones to finery verging on gaudy. The issue with the pajamas is that they aren't customisable - and therefore lack any personality. They made them as neutral as they could so it would somewhat fit every inquisitor, and (at least for me) ended up fitting none of them.
The PJs are symptomatic of the inquisitor's bland neutrality in general.
I am super fond of the mod for human females already posted though. I think it looks gorgeous, especially given what the modder had to work with.
If I had to make a neutral outfit myself, I'd keep the PJ trousers but change the boots, change the too-tight top to a more dignified tunic, probably with a slight bit of embroidery (but a slight one). Make it black and gold, because they're the inquisition standard's colours. Probably throw some sort of scarf/shawl/hood thing, possibly with fur. Something along those lines. But really, the PJs being the only choice is the issue. The fact that they're ugly as sin just adds insult to injury.