The qun says they will one day invade. Not all qunari. And how do you know Iron Bull is "atypical?" Have you met them all? I've seen several who lack that certainty of Sten and the Arishok. Ex: Tallis, Iron Bull, Gatt.
I don't think the bakers much care for what the military does but I also don't expect them to do anything if the Arishok orders an invasion. Did the soldiers balk at attacking Kirkwall despite spending years there?
And of course IB is atypical. It is the very essence of his character, that he is crossing the line into tal-vashoth, he was sent South because he was cracking under the pressure, remember?
Certainly, we have seen qunari who have doubts but only one actually abandoned the Qun and even Tallis was once sent to the kitchens precisely because she had failed to achieve certainty.
The tone of the qun has always been that the ben-hassraths mantain a very strict control and this is an extremely disciplined society united in a common goal.
I'm not saying we should kill them all but we should definitively defang them and we most certainly should not waste our tears if our enemy loses some soldiers.