Sign in to follow this  
polyfrag

Why SQL error

Recommended Posts

slicer4ever    6760

[url="http://www.gamedev.net/topic/628118-sql-error-entering-gamedev-site/"]http://www.gamedev.net/topic/628118-sql-error-entering-gamedev-site/[/url]

 

There was a more recent discussion on the topic, but I can't seem to locate it now.  But yes, depending on your time zone(for me it's midnight), ~15-30 mins every day the site is down.

Share this post


Link to post
Share on other sites
jbadams    25677

We are investigating, but so far we've been unable to find the cause of this one -- seems to be some regular task carried out by our provider, but so far it's been non-obvious.

Share this post


Link to post
Share on other sites
Michael Tanczos    5681

Here is a brief synopsis of what happens:

 

There is a period of about 5-10 minutes where our final system activity suddenly spikes and we see a HUGE IO load that pretty much brings most of the system to a halt.    During this time the SQL server cannot answer all queries and gets about 250 queries backed up and waiting to be answered.    For a brief time the IO fluctuates up and down enough to make it possible to answer some of the queries but very few of them.   What happens is that the HTTP part of the request is answered by our reverse proxy (that's the part that says the "Site is down") and sometimes by our site software (that's the part that says SQL Error).   When it's our site software the reason for the SQL Error is "Too Many Connections".   Largely the effect is caused by the reverse proxy cache being invalidated for being down more than a few minutes, the queries getting backed up in the SQL server, and the shear number of people hitting the "WTF is going on.. refresh this shit.. damn why isn't it working?!" button.    That last button slams our server because so many of the queries used to build the page aren't even being used because the page request gets abandoned.

 

TL;DR - It's caused by an as-of-yet unknown high disk IO problem (which we think may be our ISP backing up the server to be honest) caused by a period of people rapidly refreshing their browser in an attempt to help us fix the problem (while also overloading the server with too many database queries too quickly).

 

Share this post


Link to post
Share on other sites
JTippetts    12951
Did you try turning it off then back on? Check to make sure the cable is plugged in. Maybe the intarweb hamsters need faster hamster wheels. I have three goats; do you want to try sacrificing one of them?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this