So, your definition of better games is to have a bunch of sandboxes in which the AI is real AI, and everyone else is completely retarded and doesnt understand that this is what games lack, right?
If (with improved AI) the game plot still drives the NPCs with the same motives they 'precan' their ever move for now (current games with lots of hand made custom scripting) , and leave the player to be more creative in handling the situation (instead of being driven through it like a cow by a cattle prod, not being allowed (VERY OBVIOUSLY) to take even one step to the right or the left) --- THAT is so bad ???
Not exactly a 'sandbox' is it? (IS what YOU want a cattle-chute, right?)
You can still consult your 'strategy guide' to tell you how to get thru it - it might have a bit of trouble giving you a verbatum step-by-step in my kind of game though.
Also the 'real AI' as you refer to it isnt 'real AI' - just better logic that can handle what the player does more adaquately (might even be smart enough to steer the player back to the plotline when neccessary - just not as bluntly as a "You Cant Do That" message or an invisible wall you run into if you 'take one step to the right or to the left'.
As for "everyone else is completely... " -- maybe you dont realize that people here seek to make BETTER games (better AI techniques being one of the ways). There are lots of other ways but those arent what THIS section of the Forum is about.
Maybe you might read my description of the AI assisted development system that could make improving some game AI much easier -- to give us those better games.