504 Errors & Long Downtime

Aegis

These are new. But starting around 3-4 hours ago, I started receiving 503 errors when trying to access the forums (which wouldn't load at all). They'd alternate between those and simply "This page cannot be displayed". I checked again about 20 minutes ago (from my university's computers this time) and was still unable to reach the forums. Also, the main PA front page either wouldn't load as well or took upwards of a minute to display (not sure if that's related to the forums or not).

The errors seemed to have resolved themselves in the meantime (obviously or I wouldn't be posting).

Ramius

If you got the actual 503 error page, with the "nginx" text on it, then other people should have been getting it as well, at the same time. That error should never be "client side".

Can anyone verify that they also saw it 3-4 hours ago? I've been checking in every 20-30 minutes and the server seems to be doing well, but I might have missed the window of problem that Aegis saw.

Aegis

Unfortunately I didn't take a screenshot at the time, so I can't recall what the exact message on it was, but I do distinctly recall that it did have 503 error on it, since that's relatively infrequent from what I've seen.

Aegis

Okay, have just received several 504 timeout pages (showing the nginx bit as well) as of a few minutes ago. Someone in the chat thread in D&D has also noted gateway timeouts.

Ramius

ah, there they are. The message is actually:
504 Gateway Time-out
nginx/0.5.11


And yeah, this has got to be due to the fact our searches are coming from the primary DB instead of the slave db (see the other thread in this forum).

Just in the last 12 minutes, we've had a search which took 3 minutes to complete, a search which took 4.9 minutes to complete, a search which took 3.5 minutes to complete, and a search which took 57 seconds to complete. Each of those could potentially lock the database from all reads and writes.

I sent an email last night to the server folks to get our slave db back online, but until it is done we either live with these periodic timeouts, or we live with no search. (and even with no search we are susceptible to timeouts, although they should be less frequent and less severe).

Aegis

Ah, okay. Could have sworn it was a 503, sorry about that.

FyreWulff

Ramius wrote: »

ah, there they are. The message is actually:
504 Gateway Time-out
nginx/0.5.11


And yeah, this has got to be due to the fact our searches are coming from the primary DB instead of the slave db (see the other thread in this forum).

Just in the last 12 minutes, we've had a search which took 3 minutes to complete, a search which took 4.9 minutes to complete, a search which took 3.5 minutes to complete, and a search which took 57 seconds to complete. Each of those could potentially lock the database from all reads and writes.

I sent an email last night to the server folks to get our slave db back online, but until it is done we either live with these periodic timeouts, or we live with no search. (and even with no search we are susceptible to timeouts, although they should be less frequent and less severe).

IMPORTANT FYREWULFF OPINION

I say we just nix search. I mean, we've lived without it before, I think people will manage. Plus I'd rather be able to read my current threads than be able to find stuff in old ones at the moment.

darleysam

I didn't want to go making a new thread for this, but I keep getting 'database error' screens just while navigating and stuff. Anyone else getting those?

The Cat

heaps tonight

brandotheninjamaster

darleysam wrote: »

I didn't want to go making a new thread for this, but I keep getting 'database error' screens just while navigating and stuff. Anyone else getting those?

Just got a few myself, took me about 10 mins (give or take) to get back into the forums.

Varega

I've been getting database errors around 7:56 EST this morning. Haven't gotten any in a while, If it happens again I'll take a screenshot for you.

Ramius

Should be fixed, now. Sorry for the downtime folks. The database server got overloaded with connection requests, and so alphamonkey needed to reset something.

darleysam

You guys were busy celebrating the Kurt Cobain shenanigans, weren't you.

lowlylowlycook

Hmmm, I was going to say that downtime could be blamed on the Zombie infestation but maybe darleysam is right.