Cole: through the Templar path, yeah, he may have helped me defeat the demon, but... so did Mouse. Even if I went the mage route and Solas could tell me that he is not, in fact, a demon, spirits are hardly better. Spirits are driven to fulfill their own aspect. In his case, it's compassion. I cannot guarantee I can/will always satisfy any one single value or principle of a spirit to keep it happy, and compassion is hardly something that can be expected out of a group like the Inquisition.
All that said, he is actually my favorite companion. I just need a decent RP reason to recruit him.
Iron Bull: Qunari spy. Just because he is upfront with you about it does not suddenly mean he cannot steal sensitive info and sell you out (turns out, his name is "liar"). And to many people, religion is the single most important thing in their lives. 'Bull's first loyalty is the Qun, not me. I respect that, but it also means that if there is a conflict between his religion and my wishes, then I am at the disadvantage.
Also did not like the idea of him reporting back to the Qun, even if he says he won't report back anything too sensitive (yeah, okay "Hissrad"). I value subtlety, indirection, basically all the Leliana kind of things (she was my first choice for almost every War Table mission), so I see this as a leak. To his credit, Bull handles your rejection pretty graciously.
Sera I only accepted because I knew that Cole was not likely to be someone I would want to recruit, and thus, I needed an extra party member with rogue skills in case something were to happen to make Varric unavailable to me. However, I identified her as someone whom I would need to be very careful with and keep happy to avoid having problems with her, because I am in many ways the kind of elitist that Sera takes issue with. In the end, I think this dynamic ended up making me enjoy her very much. But my canon Qunari Inquisitor did not want any part of her, giving his trademarked seal-of-disapproval: "No" (interestingly enough, Sera makes a quip about Qunari and the word "no").