Their focus was no on making completely compelling villagers... there were more important things to do on a project of that size.
I've played well over 500 hours of Skyrim, and I don't see anything that makes the NPC AI good or special. They just walk around on schedules and play back pre-recorded lines. It's all marketing speak!
Speaking of which...
Follower NPC AI is among the worst ever.
That is actually a fairly ridiculous statement.
There is plenty they do very well. The problem is, the better AI gets, the less you notice it precisely because it looks normal.
Points of Interest
Sure, there are things they could have done better... but "worst ever?" Not even remotely.