Its hard to give you firaxis' reason (unless one of their employees lurk around here), but there are many reasons to choose lua over python.
The main advantages i can think of are:
1) Lua is smaller and faster. (at basic arithmetics and function calling, it loses greatly at tasks which the python standard library handles)
2) Lua has a slightly simpler syntax (no strict indentation) which makes life easier for modders, allthough i guess those who modded Civ4 are unhappy about the change as they most likely get less flexibility and have to learn a new language).
If they wanted people to write Civilization mods that turned the game into a first person shooter then yes, Python would be a superior choice, but for setting a few values and flags at runtime it just doesn't make much sense.
I don't suffer from insanity, I'm enjoying every minute of it. The voices in my head may not be real, but they have some good ideas!
I think it was due to speed. One of the original complaints about Civ 4 was it was too slow. Both Python and Lua are equivalent high level languages so flexibility wasn't the issue, imo. Lua in general runs about 4-5x faster than Python..