Historically most RTS AI has been pretty scripted. If you go back and play Starcraft/Warcraft you'll notice that when playing against a specific side, they will have 1 of maybe 3 build-up routines and will attack you at exactly the same time every game.

There's not much literature you can find around on the net about how to do an RTS AI. The best I can hint you at is divide & conquer. The way we've been going is dividing up the world as follows (actually I got this idea from some article I read on gamedev maybe 3 years ago):

UnitAI -> individual units
GroupAI -> groups of units (a drag select, an attack force, etc)
StrategicAI -> global strategic command (x soldiers attack here, y attack here, etc)

Within Strategic I've found it intuitive to break it down some more:

Economy -> farm building/finding
Production -> tech tree following / unit building / upgrades
Targeting -> find strategic targets
Tactics -> execution of tactics to capture/kill/etc Targets

Each subsystem is easily 2-3 weeks of work to get something rudimentary & robust up and running.

By dividing it up, you can break the challenge down into smaller problems that have a lot more information available as to how to solve them. UnitAI is pretty well understood and tutorialed from an FPS AI perspective.

Similarly Grouping is pretty well understood (think flocking, queuing, etc.

The strategic layer you can look towards military strategy and tactics to inform you how to pick targets and what types of tactics work for you. The biggest thing for a strategic layer is metadata (you can google for info on these terms): influence mapping, opponent analysis (what units/structures is he building, what are the weak points in his tech tree, etc), board analysis (where are my troops, where are his, where are the weak points in his line). For my game we're using some other metadata that's very game specific.

Separate the systems that collect knowledge from the systems that act on knowledge. And make it modular. For instance, create a Targeting system that has a vector of Target types each with their own heurisitc. That way if you come up with new target types you can just write little modules to find them.

All in all, a good RTS AI is no easy task. There's a lot of systems involved there and a lot of necessity for tuning the various systems to work in harmony. In general the more you can completely isolate the systems the happier you'll be.

Other advice: work in randomness. Don't always just react to the player, give your AI some initiative and some ability to randomly choose: targets, tactics, tech tree. Randomness is one of the best smoke and mirrors you'll find. Players will often mistake random actions for intelligent behavior.

Someday, when i'm not bound by NDA, i'll write a few articles about the systems I've worked on. =)

-me

The strategy and tactics scripts for Dark Reign II are available if you have the game (it must be a $10 title by now) with the unpack utility that comes with it.

Paladine's post says the rest better than I ever could so just read that :)