As my half-hearted attempt to be productive-
Any attempt to find a solution to the Chantry-Mage-Templar issue should identify what the end-states of the solution should actually be. The point of reforms of the Circle system isn't to preserve the Circle system: that's just tautological reasoning from an organizational perspective, an institution that exists to perpetuate the institution. If that's all there was to it, then there wouldn't be a point to it.
Then, and only then, once the institutional goals have been identified should policy changes be made. Policy changes should fulfill, and not subvert, the end-state: a system that seeks to at least register and track the mages, for example, should not have a policy of refusing to use phylacteries. This no more furthers the goal than an education regime that allows anyone to leave it before passing.
After that, ways to safeguard the system and keep it from changing will be needed. This means identifying weaknesses in the proposed changes, and seeing if they can be mitigated or subverted by the actions of hostile actors. Enduring institutions are planned on the expectation that people will defy them, not on the expectation that no one will have reason to defy them. If your great idea's solution to the Anders Delimma (someone who steadfastly believes it fundamentally wrong and will work to tear it down and radicalize people against it) is 'Anders wouldn't mind my system,' try again.
That's the basic helpful guidelines for creating a system. As an example, I'll do an analysis of the current system: not advocacy, but rather looking at what it attempts to do. The merits or justification are not the point of the analysis.
The Andrastian Circle System has four main goals: to prevent the rise of another mageocracy, to protect the mundanes from deliberate mage abuses, to protect everyone from accidental mage abuses (abominations), and to protect the mages from mundanes (lunch mobs and mass fear).
With the compromise that established the Circles, the agreed upon means of achieving these goals is societal segregation. Mages stay over here, mundanes over here, the division is mediated and enforced by the Chantry (with its enforcement arm of the Templars), which represented the most respected international institution for the participating land-rulers. While mundanes face some basic but largely weightless restrictions (kingdoms can not use the mages as national resources, mundanes can not access the towers and the mages at will), most of enforcement of the segregation comes by bringing outside mages into the Circles (apostates) and keeping those already in the Circle from leaving at will.
Each of the main goals has a general policy associated with achieving it, though the policies overlap and reinforce eachother. There are problems with them, but also attempted compromise solutions. Not all of these work.
Preventing the rise of a mageocracy comes via political disenfranchisement and separation from the secular power structures. Mages, who have both advantages in competitive advancement (magic advantages) and the ability to outright control political elites (blood magic) are kept out of not just the ruling class, but out of the classes that can access and reach the ruling classes. Societal segregation as a broad stroke.
The obvious issue with this is, well, mage organization and self-determination. Mages chose to go to the Circles originally, that doesn't mean they wanted to give up all rights and influence.
The attempted compromise of the Circle is in-Circle mage autonomy, in which the day-to-day management and direction of the mages is run by the mages, and an exception system in which court mages can advise national leadership and present their views and concerns. Similarly, mages can appeal to the Chantry over the Templars. Mages are not allowed to break the overall policies and restrictions of the Circle system (autonomy not meaning self-determination), but the Templars watch, not direct, Circle activities.
Protecting mundanes from hostile mages is a requirement that requires proactive, not reactive, policies. Chasing after an apostate after he kills people and flees the area is at best avenging the lost, not preventing them from being lost in the first place- this is the focus of the current system that bears remembering. Prevention, not reaction. Short of time-travel and future foresight, social division is the only reliable proactive approach to prevent such a clash. When mages are known, they are to come or be brought to the Circle and separated from the public. Once removed from the public, a mage can not harm the public.
The issue here is that mages appear outside of the circle naturally: maintaining a social division is reactive and can only be maintained by constant effort. Moreover, some people will defy the Circle and be those hostile maleficar for their own interests.
The reaction/solution to this is the Templars as a permanent expeditionary force. As much as they can they proactively go out looking for newly-manifesting mages to bring them in as soon as possible, while confronting maleficar as they are identified.
Accidental mage abuses comes in two parts: reckless/accidental usage of magic, and abominations. Both mages and mundanes are at danger to these.
For reckless/accidental magic, the danger can primarily be mitigated (though never completely resolved) by magical education. Practice, supervision, and education can remove much of the risk associated with uncontrolled magic: once a mage reaches maturity, accidental magical mishaps appear rare. Failed experiments or other issues may cause disaster, but these are failures of deliberate attempts.
For abominations, however, education can only reduce, not resolve, the risk. The source of abominations, demons, have no known or proven counter or means to eliminate, while the personal weakness that leads to an abomination is as simple as stress- an unavoidable circumstance that no system or training can remove. Very experienced mages can fall to insanity, made all the riskier by the fact that demons are deliberate actors. Though there are factors that increase the risk, there are no known factors that remove the risk: even seemingly benevolent spirits can be dangerous and drive the subject to insanity. To make things worse, there is very little preventing someone who wishes to become an abomination from becoming one: personal state of mind is the primary barrier, and it is not reliable on a system scale. Only exceptional people, as shown by exceptional accomplishments and experience, have both the chance and demonstration of will to handle extreme stresses.
For this, the mitigation for damage is containment. Education serves a role in reducing the occurrences, but education does not prevent the occurrences from happening. Abominations are something that are responded to, and the costs of the response will be proportional to the time it takes an armed response to overwhelm it. In rural settings, the cost to mundanes can be high double digits from a single abomination, as response is some time away. When the response is right down the hall, even an entire tower of abominations can be contained from harming mundanes.
The only way to minimize response time is to have the response force on standby- the only feasible way for that is the concentration of mages. Segregation isn't required for mitigation, but it synergizes very easily with it.
Protecting mages from mundanes is the less noted but one of the original basis for the Circle. The social segregation achieves this in two main forms: protection from mundane mobs, and protection from mundane elites.
The mobs is easiest, and shares with the protection of the mundanes from the mages. Separation is enforced: mages can not leave the Towers freely, but mundanes can not approach it either. Whether mundanes would want to approach a mage settlement is a question... but when mages were unorganized and still individuals, a mob overwhelming them was a real threat. Now moats, thick walls, and a small army stand between them.
Protection from mundane elites is less realized. Mages are potential assets, but also potential threats to the leaders of nations. The mage that heals your sick masses could also be an agitator and attempt to be a political rival... but unlike mobs, elites can gather the forces to overwhelm them anyway. But the mages are extremely difficult political targets due to the Chantry and Templar's military and political interference, and the Circle's role under the Chantry (itself a largely neutral body) keeps it out of national and international politics as a faction to be sided with, or against.
So, in wrap-up, the Circle has four main goals, all of which are largely met by enforcing systemic segregation between mages and mundanes. But how to keep it up?
The dilemma of who will keep the system is running with three main issues: the enforcers need the resources to support it, the legitimacy to run it on an international level, and an enduring, not circumstantial, interest in maintaining it rather than sabotaging it. A backer lacking in any of these will not be an enduring backer. At the time of its creation, the only backer capable of creating it was the Chantry.
For resources, the Chantry has a number of pillars of support. The alliance with the Orlesian Empire, the largest secular power in south-western Thedas, is a strong backer, but the support (and possible tithes) from all the Andreastian nations is relevant. Popular support and donations from the faithful. And, of course, a monopoly on the lyrium trade. No other organization in Thedas has as many revenue sources. One of the greatest restrictions of making an alternative Circle system will be 'who can afford it?'
For legitimacy, in history it has frequently been a matter of race, religion, or ideology. There is no unified racial actor interested and able to act across Thedas: the Dalish have a presence, but an antagonistic and xenophobic relationship with everyone else. They don't want to, and probably wouldn't be trusted. Similarly, there is a lack of a widespread ideology to unite nations in this manner: the closest is the Qun, which is a localized state-actor in its own right. The Grey Wardens, which are an international institution with an ideological bent, have an ideology fixated on the Darkspawn and have other issues that make using them as backers problematic. Which leaves religion, and the Chantry- the only institution in existence that (a) is supported by everyone enough to run such a system, and (
actually exists. This is the big issue with trying to secularize the Circle system and divorce it from the Chantry: there isn't a secular organization who can both afford and be respected to run the system.
As for maintaining the system as it is, this is where an enforcer organization dominated or even heavily influenced by mages runs into issues. The systems works on societal segregation: that onus is felt most by the mages, who have the greatest interest in dismantling it- they have a collective interest in weakening, not maintaining, the policies that meet the goals of the system. Mundanes, who have the most interest in the goals of the system (which protect them and their ability to claim power and political dominance), are the more reliable group to pull from. Even the sub-group to enforce it, however, will need incentives to keep the system running rather than be bribed or bullied into giving it up: those incentives crudely but practically include the monopoly of influence of letting nations request mages for various purposes, as well as a justification for the lyrium monopoly. A system that has a vested benefit in the status quo is one that can be relied upon to maintain it- morality or ethics alone can not.
So, crude and illiberal as it is, the Circle system has points to it- goals to be met, and means to meet them.
But- and here's where criticism and reform of the current system can come into play- not everything the Circle system does advances or is necessary for that goal. The system provides unintended benefits, and unnecessary costs. It does things it doesn't strictly need to do, for good and ill, and doesn't do things that it could stand to benefit from.
Here are system aspects that are not key points of the system, but aspects of it that bear consideration for good and ill.
-Apolitical mages in the established poltical system. Under the Circle system, mages are not effectively tools of the state, and are not used in international (or intranational) conflicts as such. This is a benefit of an international Circle system run by an international institution, rather than a nationalized Circle system run by the states. Being drawn into secular conflicts is also a significant risk of the mage independence movement, especially if they achieve independence via the support of national units.
-Non-magical secular education. It's... technically irrelevant to the sort of education mages need to resist demons. Literacy, liberal education, critical thinking: these are productive ways to address the need of education, but strictly not necessary. Mages are an educated populace, literate and aware of the world, in excess to not only the mundane norms, but also the system's own needs.
-A unified mage polity. While a mage identity would exist regardless, a mage polity is not as inherent: mages can be divided against eachother and less unified as a political class, as we infer from the Avaar barbarian shaaman and the Rivain seers (where the female mages are a part of a matriarchy, but male mages are not). The consolidation and organization of the Circles has permitted the rise of the mage fraternities, and the autonomy of the Circles has provided the Circles as a mage polity in and of itself. This was not necessary: the Chantry and Templars could have actively divided and pitted the mages against eachother, sabotaging unity and common identity.
-Organized magical research. The fields of mage magic and research, while limited and regulated in various degrees, is an accomplishment of the Circle systems that is not actually necessary for its goals. Past a point, further magical knowledge does not secure protection from demons or protect mundanes or mages from eachother. It just further enables the practice of magic. Allowing this is not a given: an alternative Circle system could destroy books and records and not only limit but roll back magical practice and rituals that empower mages.
-The rise of anti-magic fields of study. Templar abilities, Tranquility, and some anti-magic magics: these are not a norm, these were developments pursued and sustained by the Andrastian Circle system. They do not have known equivalents in other systems: the Tevinter mages even keep their Templar enforcers from developing and maintaining the lyrium-based anti-magic abilities. With the possible endpoint of advancing the Tranquility ritual to strip magic while leaving emotion, the Circle system has also seen the establishment of this field of supernatural anti-supernatural.
Here are some things the Circle system does that don't really benefit it. Consider it my list of reforms that would be both reasonable to demand and not detrimental to implement (within the scopes of the system).
-Disregarding the emotional health of the mages and mundanes. The Circle's security focus has left a blind spot to the human (and elven) element of mages. Anti-abomination education is focused on discipline and suppressing emotions: far less attention is given to helping mages become emotionally mature and able to cope with stress. The Circle and Templars own needless additions to the stress are likewise counterproductive.
For emotional relationships, there is an actual basis for opposing them. Strong emotional ties are an effective leverage against the primary defense against demons, the individual mind. Desire demons in particular can prey on emotional wants and ties, especially when the demon can offer something to protect or fulfill that relationship. Emotional ties outside the Circle are also an issue: outside ties are a basis for wanting escape and and support network for achieving it. For a system based on social segregation, the fewer ties between the groups the better (and this includes family ties).
But total emotional suppression, besides being unhealthy, is also unnecessary. It's throwing the baby out with the bath water: mages should be encouraged to form emotional ties inside the Circle, to support and help them there. It's a moral and practical boon: morally it can allow meaningful relationships, and practically the more ties the mage has in the system the more ties will keep them there. One of the interesting aspects of people living in authoritarian regimes is why they don't leave: friends and family are often a cause, and if a person can't take them on the escape they often won't. This isn't just friendship, but also love: families can be enabled and allowed in the Circle system, and I would even go so far as saying that bringing families from outside the system in (if they want) could also be managed and encouraged to help fulfill mage's emotional needs.
This isn't unlimited, mind you- and some of the policy implementations of maintaining the social division by keeping the mage families inside the system may frustrate those wanting more and more freedom- but the point is that mage families are compatible with the goals of the system.
Ah, but I dropped a line about the mundanes didn't I? What was that? Well, the Circle system's fixation of mental discipline, rather than mental health, has also left it a signficant issue in recognizing mental un-health. Demons are to blame for that as well, in part, but Thedas has no real comprehension of mental disorders: as far as they are aware, there are normal minds, and there are demons. Disturbed people, like the Magister's Son in DA2 (the seriel killer) have no help or understanding: they are deemed depraved, not even lunatics, because they have no demons to blame. Mundanes with real issues can not get the help they need, or even be recognized as needing it, because the cultural understanding of mental troubles is 'a strong will prevents problems.'
The field of psychology is one Thedas really, desperatly needs, for everyone. Mundanes will benefit. Mages will benefit. Demonologists and studies of the Fadewould probably benefit as well. But no one even recognizes the need because of the current fixation.
-Having the Templars constantly watching inside the Circle.
This is probably the most common and disconcerting role of the Templars in the system. It's also one of the least efficient aspects, and I say that as a very cynical person who knows something about surveillance states and how they can work.
Templar observation really doesn't do much. Not in terms of what it's supposed to: Templars aren't omnipresent enough to catch and track all the happenings, aren't effective enough to prevent the subterfuge and achievement, and almost certainly aren't trained enough to spot what they actually need to: signs of emotional volatility and instability. That field simply doesn't exist in Thedas, related to psychology. Recognizing issues is intuitive, not meticulous, and the fact that the Templars themselves are a source of anxiety and stress will only needlessly muddle the analysis.
Templars don't need to be in full battle-rattle in every room to do what they actualy do do effectively: respond to abominations. The Templar in the room is extremely unlikely to be able to prevent the abomination, or end it then and there. That's what a squad of guards down the hall, and a garrison on the ground, can do.
Templar observation should be smart. Observers should appear casual, not dressed to kill. They should try to put the room at ease. They should be approachable, though not friendly: there to make sure teachers are sticking to the approved lessons, that sedition isn't being raised vocally, that no one is preparing ambushes and barricades in the halls, and most of all just to get a feel for the public discourse and tone of the groups. And for Andraste's sake, no need to have them spying on the dorms or in their bed rooms.
That's what secret passageways behind the walls and peepholes can be used for, after all.
More seriously, overt observation is of limited usefulness. Covert observation is where it's at. Get listening devices. Get spies. And definitely get snitches- there should be so many snitches that you should never fear a group of seven mages huddled in a corner talking quietly because at least two of them should be yours.
Hey, if you're going to run a surveillance state, at least run an effective one. The less intrusive means are often the most effective.
-Cramped buildings with no outdoors.
The art of architecture is legit: building design can affect the attitudes and mental states of people, as can the availability of sunlight and sky. Circles should be tailor-made to be living areas that contain people, not containment centers that people live in. A good Circle should resemble a university campus, not a prison: dorms, grass lawns, exercise areas, water sources, avenues to walk and socialize, dedicated buildings so that your experiments in fireballs aren't being done right above the student dormitories.
Mind you, this would be expensive- the better, the more the mages would have to chip in. And it would still be a restricted area that is almost a military garrison as well: don't be surprised if the big courtyard is surrounded, if not by buildings, but impassible walls and guards with bows pointed inward.
But an restrictive environment doesn't need to feel like an oppressive environment- some people won't care if it's a nice cage, but more people will accept the gilded one. That works for keeping the system intact.