Colo Work 12/17 & 12/19

Älphämönkëy

Hi All,

I'm in the colo upgrading some other systems. If all goes well, you shouldn't notice an interruption, but just giving you guys a heads up.

If you notice problems with any of the Penny Arcade sites let me know in here.

-Alpha

UPDATE 12/19: I'm back in ur colos, upgrading ur firmwares

MKR

This is some sort of devious social experiment to see if people associate random issues with "things not going well at the colo center."

I'm on to your game.

Also, I've never seen a picture of the servers that run PA. I feel like this is a heinous oversight.

Lewisham

MKR wrote: »

Also, I've never seen a picture of the servers that run PA. I feel like this is a heinous oversight.

Khavall

I assume the

Warning: Memcache::connect() [function.Memcache-connect]: Can't connect to 172.17.17.133:11211, Connection refused (61) in [path]/includes/class_datastore.php on line 222

Fatal error: Unable to connect to memcache server in [path]/includes/class_datastore.php on line 224

Was because of this, but it looks like it's been fixed now so.

Älphämönkëy

That was my mistake. Sorry guys.

Note to self: vBulletin does not support failover caches...

As to a photo of the servers, once I'm done I'll snap a few photos.

Linksville

alphamonkey wrote: »

That was my mistake. Sorry guys.

Note to self: vBulletin does not support failover caches...

As to a photo of the servers, once I'm done I'll snap a few photos.

Staxeon

haha, got the email back Alpha. Np

Jasconius

Linksville wrote: »

alphamonkey wrote: »

That was my mistake. Sorry guys.

Note to self: vBulletin does not support failover caches...

As to a photo of the servers, once I'm done I'll snap a few photos.

I don't see any vacuum tubes, mister.

SoaL

i thought they were cardboard

Älphämönkëy

Alright! I got some good work done today.

I'm going to have to come back to finish the rest of the software upgrades tomorrow sometime.

About a week ago I got an alert telling me that our file server lost two hard drives in rapid succession. It turns out that a bad RAID controller firmware is the culprit, so I have to take things down for a bit to patch it all up. It looks like it will be Saturday night / Sunday morning. I'll keep you guys posted.

Ruckus

Also if Khoo is actually there, slap his ass and tell him SE++ says hi (kiss kiss).

MKR

Ruckus wrote: »

Also if Khoo is actually there, slap his ass and tell him SE++ says hi (kiss kiss).

The servers can't operate at such a Khool temperature, so they have to keep him away from them.

EvilBadman

Don't forget Alpha and Khoo plus server pictures. Make 'em sexy.

Buttcleft

EvilBadman wrote: »

Don't forget Alpha and Khoo plus server pictures. Make 'em sexy.

Thankyou for putting the image of people using USB slots in all the wrong ways into my head.

I'd Fuck Chuck Lidell Up

Buttcleft wrote: »

EvilBadman wrote: »

Don't forget Alpha and Khoo plus server pictures. Make 'em sexy.

Thankyou for putting the image of people using USB slots in all the wrong ways into my head.

first they're going to upgrade the firmware. then they are going to insert their cache sticks at 45 degree angles. And we know those hard drives won't fit into floppy bays so they'll just have to shove it in the Universal Serial Butt.

Älphämönkëy

I think that should be the last of the updates there. If you see anything broken, please let me know.

I'm going to clean up and assuming there are no problems, call it a night.

Henroid

alphamonkey wrote: »

I think that should be the last of the updates there. If you see anything broken, please let me know.

I'm going to clean up and assuming there are no problems, call it a night.

And what exactly were you and Robert up to, HMMM?

YamiNoSenshi

I can't help but insert an extra n in the thread title.

Thanks for all the hard work.

Henroid

YamiNoSenshi wrote: »

I can't help but insert an extra n in the thread title.

Thanks for all the hard work.

Leave Munkus out of this.

Buttcleft

wheres the server pictures

BahamutZERO

what is a colo anyway

Antimatter

http://en.wikipedia.org/wiki/Colocation_centre

A colocation centre (collocation center) ("colo") or carrier hotel is a type of data centre where multiple customers locate network, server and storage gear and interconnect to a variety of telecommunications and other network service provider(s) with a minimum of cost and complexity. Most network access point facilities provide colocation.

MKR

It's where dudes and dudettes and alphamonkeys put all their servers because running your own generator, cooling system, power grid, and network hardware sucks.

Älphämönkëy

I totally forgot to post these, sorry guys. To make up for my lateness, I'm going to include a little bit of extra content. This forum post is the very rough beginning of a series of posts I'm going to make about how we run the PA tech. I'd love to know what you guys think.

But I'm getting ahead of myself, first the photos:


This is a skewed view of our gear. I had to shoot cleverly because I can't show any other gear in this cage and I there isn't exactly a lot of space.

Top to Bottom:

• db2 - forums mysql server, failover database server for all other sites
• web8 - forums
• web5-web1 - forums, childs play charity, pax site, other "small sites"
• Gap for Networking Gear (half rack, mounted on the back)
• web7, web8 - mainsite and other "large sites"
• db1 - primary database server for everything but the forums

This is a close up of the servers you guys care about, the forum servers!

Top to Bottom:

• db2 - our newest database server, db2 sports two quad-core CPUs and 8Gb of memory
• web8 - our newest webserver, web8 sports two dual-core CPUs and 8Gb of memory
• web1-web5 are old rackable machines. They have a single one-core CPU and 4Gb of memory

I'm going to be making a more thorough writeup later, but here are the design tenants of the network:

• We run a homogenous network (FreeBSD)
• Any process that is accessible from the outside must be run inside a FreeBSD jail
  • Each jail has / mounted readonly
  • Each jail typically has an 8Mb memory disk mounted at /tmp with noexec flags to prevent code execution
  • A hardware firewall enforces the rule that jails can only talk to other jails, software firewalls restrict which jails can talk to eachother
  • Each jail should use security precautions as if it were a regular server (chroot, permission dropping, etc)
  • Jails are meant to be "chained" together, eg. SSL => Cache => Application Server => Database Server
• The entire network uses copper gigabit ethernet because it is fast and cheap
• I avoid using VLANs, I maintain two separate unmanaged gigabit switches

The design of Penny Arcade is to break everything down into small, simple components. It is a "Southwest airlines" approach. Let me give a few examples of why this works for us:

• Capacity Planning for Burst Traffic: If Childs Play becomes really big, I can spin up extra jails on other servers with very minimal overhead (around 48Mb / jail)
• Security: To get access to another jail's contents would require someone to jump through a number of hoops that really isn't worth it for a site that makes dick jokes
• Scaling: Adding additional servers is very easy: install FreeBSD host, deploy jails, profit!
• Redundancy: Everything is redundant, or simple enough to replace. Example: Replacing an unmanaged gigabit switch is really easy, replacing a managed gigabit switch is tedious.

I realize this won't work for most companies and that in some cases we directly conflict with the "enterprise" way of doing things. It works well enough for us though

Buttcleft

Thankyou Alpha.

[Deleted User]

The user and all related content has been deleted.

YamiNoSenshi

alphamonkey wrote: »

• We run a homogenous network (FreeBSD)

:^:

Lewisham

This is all super badass.

I look forward TO THE NEXT THRILLING INSTALLMENTS!