GameDev.net has the most down time of any site I regularly visit.

After the new site lunched last year I expressed my displeasure with the amount of downtime that had been constantly occurring before and especially after that bold move. I assumed that the problem would diminish with time but it didn't.

Yes, before the upgrade gamedev.net was having some trouble keeping up. However, I didn't expect it to change after the upgrade. Active topics not loading? Downtime? Check back in 10 minutes. It's fixed. No new interesting threads (in the past hour); shrug.

Now I understand that it was unforeseen hardware failure which caused this latest outage but you guys should have been prepared for this

There is a point where "preparing the unforeseen" becomes unfeasible financially...

I've noticed the same thing as Steve. It's probably because part of my job is ensuring that our servers are up as much as possible that I notice how often other sites have problems, but GDNet has by far the most downtime.

There is a point where "preparing the unforeseen" becomes unfeasible financially...

Trust me GDNet is nowhere close to the financially unfeasible range of preparing against failure unless their budget is almost non-existent.

Trust me GDNet is nowhere close to the financially unfeasible range of preparing against failure unless their budget is almost non-existent.

Do we give off an impression that we are rolling in money here or somethin? =) Almost all of our ads are generic, amazon sales are poor (not to mention their affiliate program has paid less and less each year that they develop 'exciting new payment tiers' , so we depend a lot on gdnet+ subscriptions. This site is a labor of love.

Over the last year we've had to learn how to work in a way we hadn't before.. depending on a lot of open source software and moving to MUCH cheaper cloud hosting. But with so many changes it was a first for us, but it was our only option if we wanted to be able to afford to keep running the site. Prior to 2011 we had our own rack, our own servers running Windows server operating systems.. we go from that software to a LAMP stack for the first time in our existence. 2011 I feel in some ways for us was an adapt or die year.. for real.

To be fair, I made an upgrade a few months ago that has left us with next to zero downtime since then.. this one hit us out of the blue like a nuclear bomb. Our provider actually corrupted our file system by running fsck on the parent server we are on without backing us up first. Their response? "We're sorry.. " followed by our options.

I fully expect, barring some fluke like yesterday morning, to have next to no downtime in the forseeable future with this setup however. Though what we do when our provider literally destroys us.. I don't know. It wasn't intentional, but wow. I really thought we would be good.

Are you guys running a virtual machine on their server? I don't understand how they could have screwed your filesystem up with a fsck, unless the data portion is mapped to a volume on one of their SANs or something. Either way, that's unacceptable on their part.

I help run the IT shop for the computer science department at my University, so I understand what it's like to have unforeseen hardware failures. It's the worst feeling ever! I'm glad your backups worked. I'm always nervous that the backups will be corrupt.

Zach.

I say we have a gamedev fundraiser! if funds are an issue.

A lack of funds in a community of developers. Is it just me or does that sound weird?

I usually don't notice the downtimes, most of them take 5-10m to be resolved.

Are you guys running a virtual machine on their server? I don't understand how they could have screwed your filesystem up with a fsck, unless the data portion is mapped to a volume on one of their SANs or something. Either way, that's unacceptable on their part.

Yep, and the corruption occurred somehow on the host file system that housed our virtual server. Because of the size of our instance we were the only one on that particular server.. so when they ran fsck on our end we saw all sorts of screwed up stuff. The apache config, for example, opened up as some random PHP file from our website.

I say we have a gamedev fundraiser! if funds are an issue.

Well we tried a donation run back in 2010 and honestly the response was barely lukewarm. Perhaps we didn't grovel enough or do a good enough job getting the word out although it was a posted forum announcement for several months. Fundraising is something that does require effort though and perhaps we just didn't put enough into it the first time around.

[quote name='ZBethel' timestamp='1325947474' post='4900343']
Are you guys running a virtual machine on their server? I don't understand how they could have screwed your filesystem up with a fsck, unless the data portion is mapped to a volume on one of their SANs or something. Either way, that's unacceptable on their part.

Yep, and the corruption occurred somehow on the host file system that housed our virtual server. Because of the size of our instance we were the only one on that particular server.. so when they ran fsck on our end we saw all sorts of screwed up stuff. The apache config, for example, opened up as some random PHP file from our website.
[/quote]

Unbelievable. That's the whole point of running a virtual server, to have a buffer of security between the guest and a hardware failure. I'm assuming they run a VMWare cluster of some kind, in which case they may have had a SAN failure or a disk array failure. At any rate, they should have had your stuff backed up for you at the virtual machine level.

A lack of funds in a community of developers. Is it just me or does that sound weird?

I usually don't notice the downtimes, most of them take 5-10m to be resolved.

To be honest, a fundraiser would be too short term of a solution to really have much of an impact on the operation of the site. We cut our costs dramatically in 2011 and have at least stopped accumulating a ton of debt and are paying it down. At one point the interest on our debt was matching our hosting costs, and 2012 was the year we would close shop.

We are trying really hard this year to expand the marketplace - because that seems like an ideal way to fund the site. We do at minimum a 70/30 split with any publisher where the publisher makes the 70% on each item. If that were to be hugely successful we could both thrive as a site AND really help out the artists/musicians out there that have become part of our community. We explored the idea of doing it with software, but a lot of developers have special requirements for generating license keys and tracking active licenses.

Now, our goal is to start working with the thousands of sites out there who are publishing a tutorial here and there and trying to get the best of them either linked to or posted on our site. Right now *anyone* can post an article.. no restrictions. Click on the resources tab and if the 'add article' button is there.. you can easily add articles. We really are a site that cares first and foremost about the information, whether it is the forums, journals, or tutorials. The best way to help the site is to give back to the community with journals or articles.. information sharing is what it's all about. =)

I could be thinking too far future (or out my butt) but maybe GDNet needs to advertise the Marketplace a bit more or start looking at making Workshops profitable. But I admit that we, the members, probably haven't been doing enough to support you. Plus with an upgrade and technology change at the scope you have done, I don't see the amount or frequency of downtime to be such an issue. By middle of this year, these sorts of things should be nigh eliminated/preventable.

With that said, I'll go buy a GDNet membership....

If you're running on a standard LAMP stack on a single server, why not go with Amazon Web Services? It's very economical, and you get some incredible features for when crashes do occur (and no service is immune to hardware failures, so it should be planned for).

You could probably run this entire site on one Extra Large sized instance (15GB RAM, quad-core, about 100mbps of available network bandwidth), which including bandwidth, redundant EBS storage, hourly backups to multiple data centers, and so forth would come out below $1,000 per month. If you pay up-front for a "reserved" instance, that price drops considerably. You might even get by with a server half that size, depending on the real site load, which obviously I don't know the details on.

Once everything is up and running, you take an image of the server, and then if you ever need to replace it, you can do so in a matter of 2-3 minutes, plus about 2 more minutes to restore the data partition from a recent backup. The process to upgrade to a larger system if needed is even shorter: stop the instance, change its size, restart it. The whole thing takes 2 minutes at worst.

These days, AWS is even PCI Level 1 compliant, so you can feel free to take credit card payments there.

It's pretty straightforward to get something set up over there, and I'd be happy to help. My company LucidChart is all set up on AWS, so I have pretty solid experience setting up systems there. Let me know if you'd like me to help, I'd be happy to help for free. GameDev helped me out a lot a decade ago, and I'm happy to help back :-)

If you're running on a standard LAMP stack on a single server, why not go with Amazon Web Services? It's very economical, and you get some incredible features for when crashes do occur (and no service is immune to hardware failures, so it should be planned for).

You could probably run this entire site on one Extra Large sized instance (15GB RAM, quad-core, about 100mbps of available network bandwidth), which including bandwidth, redundant EBS storage, hourly backups to multiple data centers, and so forth would come out below $1,000 per month. If you pay up-front for a "reserved" instance, that price drops considerably. You might even get by with a server half that size, depending on the real site load, which obviously I don't know the details on.

Once everything is up and running, you take an image of the server, and then if you ever need to replace it, you can do so in a matter of 2-3 minutes, plus about 2 more minutes to restore the data partition from a recent backup. The process to upgrade to a larger system if needed is even shorter: stop the instance, change its size, restart it. The whole thing takes 2 minutes at worst.

These days, AWS is even PCI Level 1 compliant, so you can feel free to take credit card payments there.

It's pretty straightforward to get something set up over there, and I'd be happy to help. My company LucidChart is all set up on AWS, so I have pretty solid experience setting up systems there. Let me know if you'd like me to help, I'd be happy to help for free. GameDev helped me out a lot a decade ago, and I'm happy to help back :-)

Honestly, I don't think they are spending $1000/mo right now after hearing how incompetent their hosting service was. I was spending $5/mo and got better service than that.

Honestly, I don't think they are spending $1000/mo right now after hearing how incompetent their hosting service was. I was spending $5/mo and got better service than that.

Fair enough. That said, AWS seems to give better bang for your buck in almost every category, regardless of budget, at the expense of them expecting you to be able to manage installing and managing your own software on the boxes. For example, a "large" server has 7.5GB of RAM, plus a fairly powerful dual-core processor, and costs about $250/mo for the server. Add 100GB of persistent data storage that is automatically replicated into multiple data centers, and it's an extra $10/mo. Bandwidth is purely metered, and starts at $0.15/GB--so say they do 100GB/mo of bandwidth, and it's another $15. Then say they use snapshots of their data partition as a backup mechanism. Snapshots are incremental, and you only get charged for the size of the incremental backups, so expect another $20-50/mo, depending on how often and deeply the data set changes.

Now say they're only going to run that one server, and they'll be running it 100% of the time. They do a 1-year "heavy utilization" reserved instance. They pay $1105 up front, and that $250/mo bill for the server itself drops down to $57/mo, which brings the per-month amortized cost to (1105 + 57*12) / 12 ~= $150/mo. Add in all the extra little charges, and you're still talking about less than $250/mo total hosting cost.

You can't get a better total deal for $250 anywhere. My $1000 was meant as a ceiling, if they need an extra large server and tons of bandwidth and data storage.

Honestly, I don't think they are spending $1000/mo right now

Our hosting cost is quite a bit less than $1000/month. I think at $1000/month you could look in the world of cloud hosting at having a load balancer, separate web front-ends, and perhaps a replicated database server.

We have one primary server with 8 dedicated CPU cores, 32GB of ram, and 1.5TB of space.

[quote name='SteveDeFacto' timestamp='1325966225' post='4900420']
Honestly, I don't think they are spending $1000/mo right now after hearing how incompetent their hosting service was. I was spending $5/mo and got better service than that.

Fair enough. That said, AWS seems to give better bang for your buck in almost every category, regardless of budget, at the expense of them expecting you to be able to manage installing and managing your own software on the boxes. For example, a "large" server has 7.5GB of RAM, plus a fairly powerful dual-core processor, and costs about $250/mo for the server. Add 100GB of persistent data storage that is automatically replicated into multiple data centers, and it's an extra $10/mo. Bandwidth is purely metered, and starts at $0.15/GB--so say they do 100GB/mo of bandwidth, and it's another $15. Then say they use snapshots of their data partition as a backup mechanism. Snapshots are incremental, and you only get charged for the size of the incremental backups, so expect another $20-50/mo, depending on how often and deeply the data set changes.

Now say they're only going to run that one server, and they'll be running it 100% of the time. They do a 1-year "heavy utilization" reserved instance. They pay $1105 up front, and that $250/mo bill for the server itself drops down to $57/mo, which brings the per-month amortized cost to (1105 + 57*12) / 12 ~= $150/mo. Add in all the extra little charges, and you're still talking about less than $250/mo total hosting cost.

You can't get a better total deal for $250 anywhere. My $1000 was meant as a ceiling, if they need an extra large server and tons of bandwidth and data storage.
[/quote]
A move will only present more down time. Perhaps they should just keep their current service for the next year and hope this was a one time event. If something like this happens again this year I strongly recommend transferring to a new and larger host.

Yes most downtime by a longshot, its always been thus though, even 10 years ago (ironic since its a computer focused website).
Ive often got the page with a MS server error or something on it, I did suggest going to apache in ~2004 by was assured things would improve in future

So are we saying that MS servers are just inherently unstable?