Yeah, it's far from trivial. But it's also an area of game programming that hasn't really evolved during the past 15 years, as far as mainstream shooters go. I think it might only take one really popular, groundbreaking game that showed others what can be achieved by focusing on good AI...
I guess the popularity of multiplayer is one major reason for the current situation. Why would a game need AI when you can fight actual human beings instead?
I do blame console multiplayer for a chunk of that. I also blame console hardware for being generally under powered and AI being the easiest to "fake it" with far less processor intensive scripting tricks. Especially when trying to squeeze every drop of performance. Which is now coming back to bite many recent games in the rear, because the "people" in these gorgeous worlds are acting like they've had large chunks of their brains removed (which isn't far from the truth).
Sadly you can't market good AI the way you can market "Glitter". Like VR, Touchscreens, and Haptics its part of the "experience". And we feel it when its "wrong", when units don't behave appropriately for the environment, or perform very "gamey" behaviors.
It's why I deeply fear for Star Wars Battlefront. While the AI in Battlefront wasn't the greatest it created a very different game play experience. Dice is showing off "Glitter" and highly fidelity CG reproductions of movie assets. They aren't showing us the AI Bots, the Galaxy Conquest Mode, and other things that made Battlefront more than a Battlefield knockoff with a Star Wars skin.
IMO the best showcase as to why AI development is important is the DragonBall Z League (
) which puts teams of AI vs AI. "Coaches" of the teams customize different settings of the AI before a match, but once its running its hands off and its all down to the computer.