Penny Arcade Forums› H.Q. Reception Desk
The new forums will be named Coin Return (based on the most recent vote)! You can check on the status and timeline of the transition to the new forums here.
The Guiding Principles and New Rules document is now in effect.

Colo Work 12/17 & 12/19

ÄlphämönkëyÄlphämönkëy Registered User regular
edited December 2009 in H.Q. Reception Desk
Hi All,

I'm in the colo upgrading some other systems. If all goes well, you shouldn't notice an interruption, but just giving you guys a heads up.

If you notice problems with any of the Penny Arcade sites let me know in here.

-Alpha

UPDATE 12/19: I'm back in ur colos, upgrading ur firmwares

Älphämönkëy on

Posts

  • MKRMKR Registered User regular
    edited December 2009
    This is some sort of devious social experiment to see if people associate random issues with "things not going well at the colo center."

    I'm on to your game.

    Also, I've never seen a picture of the servers that run PA. I feel like this is a heinous oversight.

    MKR on
    one of those newsed letters - a blue sky
  • LewishamLewisham Registered User regular
    edited December 2009
    MKR wrote: »
    Also, I've never seen a picture of the servers that run PA. I feel like this is a heinous oversight.

    Lewisham on
  • KhavallKhavall British ColumbiaRegistered User regular
    edited December 2009
    I assume the
    Warning: Memcache::connect() [function.Memcache-connect]: Can't connect to 172.17.17.133:11211, Connection refused (61) in [path]/includes/class_datastore.php on line 222

    Fatal error: Unable to connect to memcache server in [path]/includes/class_datastore.php on line 224
    Was because of this, but it looks like it's been fixed now so.

    Khavall on
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2009
    That was my mistake. Sorry guys.

    Note to self: vBulletin does not support failover caches...

    As to a photo of the servers, once I'm done I'll snap a few photos.

    Älphämönkëy on
  • LinksvilleLinksville Registered User regular
    edited December 2009
    alphamonkey wrote: »
    That was my mistake. Sorry guys.

    Note to self: vBulletin does not support failover caches...

    As to a photo of the servers, once I'm done I'll snap a few photos.

    Itdzr.jpg

    Linksville on
  • StaxeonStaxeon Buffalo, NYRegistered User regular
    edited December 2009
    haha, got the email back Alpha. Np :)

    Staxeon on
    Invisible nap is the best nap of all time!
    No man should have that kind of power.    (Twitter)
  • JasconiusJasconius sword criminal mad onlineRegistered User regular
    edited December 2009
    Linksville wrote: »
    alphamonkey wrote: »
    That was my mistake. Sorry guys.

    Note to self: vBulletin does not support failover caches...

    As to a photo of the servers, once I'm done I'll snap a few photos.

    Itdzr.jpg

    I don't see any vacuum tubes, mister.

    Jasconius on
    this is a discord of mostly PA people interested in fighting games: https://discord.gg/DZWa97d5rz

    we also talk about other random shit and clown upon each other
  • SoaLSoaL fantastic Registered User regular
    edited December 2009
    i thought they were cardboard

    SoaL on
    DKFA7.gif
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2009
    Alright! I got some good work done today.

    I'm going to have to come back to finish the rest of the software upgrades tomorrow sometime.

    About a week ago I got an alert telling me that our file server lost two hard drives in rapid succession. It turns out that a bad RAID controller firmware is the culprit, so I have to take things down for a bit to patch it all up. It looks like it will be Saturday night / Sunday morning. I'll keep you guys posted.

    Älphämönkëy on
  • fightinfilipinofightinfilipino Angry as Hell #BLMRegistered User regular
    edited December 2009
    thanks for all the hard work alpha! (and tell Tycho to stop hosting all that porn, we've seen PATV, we know what's on there)

    fightinfilipino on
    ffNewSig.png
    steam | Dokkan: 868846562
  • RuckusRuckus Registered User regular
    edited December 2009
    Also if Khoo is actually there, slap his ass and tell him SE++ says hi (kiss kiss).

    Ruckus on
  • MKRMKR Registered User regular
    edited December 2009
    Ruckus wrote: »
    Also if Khoo is actually there, slap his ass and tell him SE++ says hi (kiss kiss).

    The servers can't operate at such a Khool temperature, so they have to keep him away from them.

    MKR on
    one of those newsed letters - a blue sky
  • EvilBadmanEvilBadman DO NOT TRUST THIS MAN Registered User regular
    edited December 2009
    Don't forget Alpha and Khoo plus server pictures. Make 'em sexy.

    EvilBadman on
    FyreWulff wrote: »
    I should note that Badman is fucking awesome
    XBL- Evil Badman; Steam- EvilBadman; Twitter - EvilBadman
  • ButtcleftButtcleft Registered User regular
    edited December 2009
    EvilBadman wrote: »
    Don't forget Alpha and Khoo plus server pictures. Make 'em sexy.

    Thankyou for putting the image of people using USB slots in all the wrong ways into my head.

    Buttcleft on
  • fightinfilipinofightinfilipino Angry as Hell #BLMRegistered User regular
    edited December 2009
    Buttcleft wrote: »
    EvilBadman wrote: »
    Don't forget Alpha and Khoo plus server pictures. Make 'em sexy.

    Thankyou for putting the image of people using USB slots in all the wrong ways into my head.

    Khoo and Alpha mounting their drives. you know you like it.

    fightinfilipino on
    ffNewSig.png
    steam | Dokkan: 868846562
  • I'd Fuck Chuck Lidell UpI'd Fuck Chuck Lidell Up Registered User regular
    edited December 2009
    Buttcleft wrote: »
    EvilBadman wrote: »
    Don't forget Alpha and Khoo plus server pictures. Make 'em sexy.

    Thankyou for putting the image of people using USB slots in all the wrong ways into my head.

    first they're going to upgrade the firmware. then they are going to insert their cache sticks at 45 degree angles. And we know those hard drives won't fit into floppy bays so they'll just have to shove it in the Universal Serial Butt.

    I'd Fuck Chuck Lidell Up on
  • fightinfilipinofightinfilipino Angry as Hell #BLMRegistered User regular
    edited December 2009
    I'd Fuck Chuck Lidell Up wrote: »
    Buttcleft wrote: »
    EvilBadman wrote: »
    Don't forget Alpha and Khoo plus server pictures. Make 'em sexy.

    Thankyou for putting the image of people using USB slots in all the wrong ways into my head.

    first they're going to upgrade the firmware. then they are going to insert their cache sticks at 45 degree angles. And we know those hard drives won't fit into floppy bays so they'll just have to shove it in the Universal Serial Butt.

    i think they're doing all these things now cause damn the forums be fucked up now.

    fightinfilipino on
    ffNewSig.png
    steam | Dokkan: 868846562
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2009
    I think that should be the last of the updates there. If you see anything broken, please let me know.

    I'm going to clean up and assuming there are no problems, call it a night.

    Älphämönkëy on
  • HenroidHenroid Mexican kicked from Immigration Thread Centrism is Racism :3Registered User regular
    edited December 2009
    alphamonkey wrote: »
    I think that should be the last of the updates there. If you see anything broken, please let me know.

    I'm going to clean up and assuming there are no problems, call it a night.

    And what exactly were you and Robert up to, HMMM?

    Henroid on
  • YamiNoSenshiYamiNoSenshi A point called Z In the complex planeRegistered User regular
    edited December 2009
    I can't help but insert an extra n in the thread title.

    Thanks for all the hard work.

    YamiNoSenshi on
  • HenroidHenroid Mexican kicked from Immigration Thread Centrism is Racism :3Registered User regular
    edited December 2009
    YamiNoSenshi wrote: »
    I can't help but insert an extra n in the thread title.

    Thanks for all the hard work.

    Leave Munkus out of this.

    Henroid on
  • ButtcleftButtcleft Registered User regular
    edited December 2009
    wheres the server pictures

    Buttcleft on
  • BahamutZEROBahamutZERO Registered User regular
    edited December 2009
    what is a colo anyway

    BahamutZERO on
    BahamutZERO.gif
  • AntimatterAntimatter Devo Was Right Gates of SteelRegistered User regular
    edited December 2009
    http://en.wikipedia.org/wiki/Colocation_centre
    A colocation centre (collocation center) ("colo") or carrier hotel is a type of data centre where multiple customers locate network, server and storage gear and interconnect to a variety of telecommunications and other network service provider(s) with a minimum of cost and complexity. Most network access point facilities provide colocation.

    Antimatter on
  • MKRMKR Registered User regular
    edited December 2009
    It's where dudes and dudettes and alphamonkeys put all their servers because running your own generator, cooling system, power grid, and network hardware sucks.

    MKR on
    one of those newsed letters - a blue sky
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2009
    I totally forgot to post these, sorry guys. To make up for my lateness, I'm going to include a little bit of extra content. This forum post is the very rough beginning of a series of posts I'm going to make about how we run the PA tech. I'd love to know what you guys think.

    But I'm getting ahead of myself, first the photos:

    200912-servers-all.jpg
    This is a skewed view of our gear. I had to shoot cleverly because I can't show any other gear in this cage and I there isn't exactly a lot of space.

    Top to Bottom:
    • db2 - forums mysql server, failover database server for all other sites
    • web8 - forums
    • web5-web1 - forums, childs play charity, pax site, other "small sites"
    • Gap for Networking Gear (half rack, mounted on the back)
    • web7, web8 - mainsite and other "large sites"
    • db1 - primary database server for everything but the forums

    200912-servers-forums.jpg
    This is a close up of the servers you guys care about, the forum servers!

    Top to Bottom:
    • db2 - our newest database server, db2 sports two quad-core CPUs and 8Gb of memory
    • web8 - our newest webserver, web8 sports two dual-core CPUs and 8Gb of memory
    • web1-web5 are old rackable machines. They have a single one-core CPU and 4Gb of memory

    I'm going to be making a more thorough writeup later, but here are the design tenants of the network:
    • We run a homogenous network (FreeBSD)
    • Any process that is accessible from the outside must be run inside a FreeBSD jail
      • Each jail has / mounted readonly
      • Each jail typically has an 8Mb memory disk mounted at /tmp with noexec flags to prevent code execution
      • A hardware firewall enforces the rule that jails can only talk to other jails, software firewalls restrict which jails can talk to eachother
      • Each jail should use security precautions as if it were a regular server (chroot, permission dropping, etc)
      • Jails are meant to be "chained" together, eg. SSL => Cache => Application Server => Database Server
    • The entire network uses copper gigabit ethernet because it is fast and cheap
    • I avoid using VLANs, I maintain two separate unmanaged gigabit switches

    The design of Penny Arcade is to break everything down into small, simple components. It is a "Southwest airlines" approach. Let me give a few examples of why this works for us:
    • Capacity Planning for Burst Traffic: If Childs Play becomes really big, I can spin up extra jails on other servers with very minimal overhead (around 48Mb / jail)
    • Security: To get access to another jail's contents would require someone to jump through a number of hoops that really isn't worth it for a site that makes dick jokes
    • Scaling: Adding additional servers is very easy: install FreeBSD host, deploy jails, profit!
    • Redundancy: Everything is redundant, or simple enough to replace. Example: Replacing an unmanaged gigabit switch is really easy, replacing a managed gigabit switch is tedious.

    I realize this won't work for most companies and that in some cases we directly conflict with the "enterprise" way of doing things. It works well enough for us though :wink:

    Älphämönkëy on
  • ButtcleftButtcleft Registered User regular
    edited December 2009
    Thankyou Alpha.

    Buttcleft on
  • CyvrosCyvros Registered User regular
    edited December 2009
    That's really cool, alpha.

    My brother works at an ISP and I IMed him a link to this page, calling it "server porn". Only just then did I notice that some of the equipment's by "Rackable Systems".

    Teehee.

    Cyvros on
  • YamiNoSenshiYamiNoSenshi A point called Z In the complex planeRegistered User regular
    edited December 2009
    alphamonkey wrote: »
    • We run a homogenous network (FreeBSD)

    :^:

    YamiNoSenshi on
  • LewishamLewisham Registered User regular
    edited December 2009
    This is all super badass.

    I look forward TO THE NEXT THRILLING INSTALLMENTS!

    Lewisham on
  • fightinfilipinofightinfilipino Angry as Hell #BLMRegistered User regular
    edited December 2009
    so what i gathered from all that is we're all really in jail anyways.



    (thanks alpha for this and all the hard work!)

    fightinfilipino on
    ffNewSig.png
    steam | Dokkan: 868846562
Sign In or Register to comment.
Penny Arcade Forums› H.Q. Reception Desk