Depends. Are Halla suitable for crossing deserts? Do Bronto find mountain climbing difficult? Would an Elf ride his mount to death just to perpetuate a stereotype? I have no problems with Elves choosing to ride Halla when it's a viable option. I think the idea that for some reason a Halla couldn't be ridden by a human, preposterous.
If the elves used to ride halla into combat, and we have seen concept art of what looks like a (young) halla being ridden on a very craggy mountain, I have no reason to think that they aren't versatile. Brontos being used underground, in mines, as draft animals, etc. I would suspect that they're also pretty reliable. There's a video showing what looks like brontos by a watering hole in the desert. I could easily see them in and around mountains, considering the dwarves handle them.
Halla seem to be semi-sacred to the elves. So, while I can see a human who befriended the elves maybe getting access to a halla as a mount, they'd be too small for a qunari to ride and maybe not sturdy enough for a dwarf (and probably too tall for the average dwarf). Even then it's unlikely the elves would allow it, since they seem to treat halla as people in their own right. So yes, it does make sense that not just anyone can hop on a halla and ride off. The Dalish halla keepers seem to have actual honest-to-goodness conversations with their halla, if DA:O is anything to go by. And that perhaps puts them above even the ridiculously smart mabari hounds in intelligence. So it wouldn't just be the human's or the elves' choice in whether you got to hop on a halla and ride off, but also the choice of the halla, unless you enjoy having a sore rear end and a trampled everything.
My point being, anyone can ride a horse, but racial mounts should at the very least require a quest that would "earn" that mount. In the case of elves, you'd have to show you could respect that halla as they do. In the case of dwarves, you'd need to show you could subdue a bronto since they can be foul-tempered. Etc. And for that matter perhaps there would be some special horse that normally only humans could ride (or perhaps not, to make the other races a bit more unique).