Well, we didn't want to switch our hosting provider so I don't know about prices in other places, but it was much cheaper to rent a dual opteron 244 than two P4 by about a 100 USD a month. Having a dual processor isn't just a question of running apache and mysql, our server can take advantage of a mutli processor system while it's not yet ready for clustering.

Edit: My bad, I didn't get the "cheap" part :p. It just might've been cheaper but the fact that our player base keeps growing played a part in that decision too. We wanted lots of room for growth before we had to go trough another server change and all the troubles involved.

I understand. When you lease a new P4 at one of those places, you also lease more networking bandwidth, which you might not be using -- but you're still paying for it.

In some places, if your two machines talk to each other, that counts as "bandwidth" that you're "using". Watch out for that! One of the reasons why a cage at a co-lo is better in the long run.

Anyway, when it comes to not having to move servers for a long time, I'd assume that a "plug in as you scale" architecture is the ultimate for that -- you don't need to "move" anything, just "add", which is usually simpler to do without re-DNS-ing or anything.

That's a very good point about local traffic between our two machines, I hadn't thought of that. We'd probably have known when we started expanding into a cluster but that's not a very nice way to know, now is it =).