1)Would bard2/barbarian2/rogue1 human get xp penalty? What about Bard5/barbarian4/rogue3?
None of those examples get an XP penalty, though the latter may have got it at some point, depending on the order it's been levelled. In the case of humans and half-elves, the base class with most levels (the favoured class in the case of the other races) is ignored as far as XP penalties are concerned, and the level difference between ther other base classes must not be more than one level.
For example, let's say bard is your main class. Bard 10/barbarian 1/rogue 1 doesn't get a penalty, nor it does bard 10/barbarian 2/rogue 1. Bard 10/barbarian 3/rogue 1does, however, since barbarian and rogue are two levels away from each other in that case.
Is bard/barbarian/rogue viable class?
It's viable in the sense that you can finish the game with it, but it's not a good combination if you want to split your levels equally in the three classes. Barbarian 1/rogue 1 or 2 and the rest in bard should be good, but that's just a bard with some extras.
It depends on the level range of the module too. Some things work well at low levels but not in epic, and vice versa.
Bard/barbarian and rogue/wizard or bard/rogue and some fighter class or bard and cleric?
Well, you can probably manage with anything, but having a cleric around is always good, and both the cleric and the bard can benefit from each other's buffs. If playing at low levels, which is the case for most modules, fighter is another good option, but a team consisting of a bard and rogue/wizard may be a bit too squishy.