If I think about this, every roleplaying game (a tabletop or video game) is to its core is nonsensical and fundamentally breaks roleplaying.
As an example, how do you feel about two very common RPG mechanics like experience points and hit points? The principle is not that different from let's say special ammo abilities or AD&D's weapon restrictions. All these are just very abstract, simplified and ambiguous means to model otherwise very complex "real world" phenomenona or representations of them. There are differences and some mechanics achieve their goal better than others, but regardless you are going to bump into constraints, incoherences and varying degree of abstractness, if you play RPGs.
Abstractions are fine as long as they're internally consistent. Hit points, is particular, work really well for me, if they model damage the same way for everyone (as they do in tabletop RPGs).
And many game systems will even make explicit what XP actually represent, allowing the actual acquisition of skills to be done off-screen.
It's quite often just sort of reversed cherry picking when proposedly "a game world doesn't make sense". All roleplaying games are based to kind of "an understanding" between participants, about what and how a game is trying to achieve or "tell". Then it's about particulars on what to focus and on what kind of abstraction level there is. Like modeling a trauma or learning with a simple arithmetic is not really that different from preventing magic-users from using swords. This is supposed to represent a time and effort it takes for anybody to learn magic, which excludes them from learning other skills. Also it has secondary function to balance classes between each other, which isn't rooted in a game lore exactly, but neither does a lore contradict that player classes shouldn't be balanced compared to each other. Even if it isn't explicitly stated anywhere. Also it's allegorical to a real world myths and literary traditions, which often set a magic as opposing force or at least contradicting one to a martial skills.
If the lore supports the mechanics, this can work fine. I'm reminded of L.E. Modesitt's ChaosWar series of books, wherein Order Mages cannot even physically touch weapons without becoming violently ill.
You cannot even wield a sword, that's true, but neither can you pick up a rock from a street or set a house on fire with a fireball, which can burn armored men into cinders. Actually if rules should reflect lore absolutely accurately, there shouldn't be houses or any buildings at all nor could they be destroyed by any means. Rules and mechanics omit everything related to buildings totally, but in case of locked doors which can be picked open or forced open. Immersion totally ruined.
1st edition AD&D contained extensive rules relating to construction, and each different material had its own saving throw table. Is it a wooden structure? Save vs. Fire. Oh, it was a
Fireball? Save vs. Magical Fire.
And if there simply aren't rules for it, that's fine too, as long as what we see isn't internally inconsistent. If the system contains no rules for buildings, then any behaviour of buildings is fine. But not any combination of behaviours; if two seemingly identical structures are subjected to relevantly similar circumstances, and they behave differently, and reason for that needs to be knowable.
I don't say your arguments wouldn't have any merit, that there couldn't be incoherence or lack of simulating which is harmful for a RPG, but to be honest you take this "it's not roleplaying"-thing to an absurd level.
I follow the reasoning wherever it leads.