304 error at 2:30am cst January 29th.

Last Son

At 2:30cst January 29th I was getting a 304 error:
"Forbidden

You don't have permission to access / on this server."

while trying to access the front page, trying to get to the forums resulted in just a blank white page. My connection to the rest of the internet was uninterrupted.

After about 10 minutes the front page returned to normal but attempts to reach the forums resulted in the error:

"Warning: Memcache::connect() [function.Memcache-connect]: Can't connect to 172.17.17.131:11211, Operation timed out (60) in /includes/class_datastore.php on line 307

Fatal error: Unable to connect to memcache server in /includes/class_datastore.php on line 309"

Its now been about 18minutes and things seem to be back to normal.

Thanatos

This would be the downtime per the announcement Alphamonkey made about the downtime.

Älphämönkëy

That was me. I was doing some planned work tonight and that was a side effect (one of our webservers died in a particularly exciting way)

Tech details:

• Stale NFS handles will cause FreeBSD 7.1 clients to crash
• Stale NFS handles will still be stale after the server reboots
• Simply restarting nfsclient is not enough, one must manually umount -f a bad mount point
• A load balancing algorithm that is based on speed and server load must check for webpage correctness, otherwise a single malfunctioning server will get to the top of the load balancing priority queue (because it sure can serve errors quickly)
• 172.17.17.131 is a virtual address of the webserver that freaked out, vBulletin did not handle it being down gracefully