If I understood it correctly, it was not the populace as a whole that the mages feared but the original Inquisition. They sought the protection of the Chantry because they were the only ones who the Inquisition might listen to. The Chantry tolerated the mages and they would appear to have had limited interaction with the community at large, even if they were largely confined to the Chantry precincts. Then mages objected to the limitations placed on their use of magic, which would appear to have been limited to healing and lighting the braziers in the chantries, etc. So they did a deal that would allow them to use their magic freely within a specific framework overseen by Inquisition member, now renamed the Templar Order. Part of the deal involved them occupying Circle buildings located away from the main centres of population, which is why I find the White Spire and the Gallows contrary to that idea.
You can see the merits of remote towers. When the Circle was overrun by demons in Ferelden, the Templars were able to contain it entirely within the building. When Meredith attack the Circle in the Gallows we had mayhem on the streets of Kirkwall. This was particularly odd since Orsino had been complaining to us how they were being kept confined to their rooms, yet both he and many supporting Enchanters were able to leave the Gallows entirely and end up in Lowtown on the way to the Chantry. It is laughable how easy it is to get in and out of both the Gallows and White Spire without the Templars being aware. They clearly had no imagination because it couldn't have been that hard to systematically make a check of all lower areas, tapping on walls, etc, to discover areas of weakness, secret passages and the like, and then seal them up. After all it is not as if these buildings had only recently been occupied. Regular maintenance checks could then have been carried out, say once a year, to ensure that new compromises in security had not occurred.
My main reason for not supporting the Templars in their current form is that on the whole they appear to have been absolute idiots and for the most part useless at performing the job they are meant to. Even in the Ferelden Tower, which seemed better run than others we have encountered, the First Enchanter was able to form an alliance with the Architect under their noses and then subsequently Uldred was able to completely corrupt the tower with blood magic, when the only person they seemed concerned with was Jowan. Then in the demo recently, Templars in the Hinterlands (so presumably from Ferelden) are incapable of telling the difference between a farming tool and a mage staff. Just don't get me started on how ineffective they were in Kirkwall. So much for the expertise of our magical police force.