Recent downtime

245

Posts

  • EndEnd Registered User regular
    edited December 2010
    Henroid wrote: »
    Wow. So when those 2 jailed people were posting, were they inadvertently causing problems?

    Heh.

    Naw, jails is a FreeBSD thing. It's like a lightweight partial virtualization (similar but not quite like VMWare, Parallels, etc), but it only works within the context of FreeBSD (the OS that runs the forums). I'm not sure that was that great of an explanation, but at worst: it's a server thing with making one server look more like multiple.

    End on
    I wish that someway, somehow, that I could save every one of us
    zaleiria-by-lexxy-sig.jpgsteam~tinythumb.png
  • HenroidHenroid Radio Demon Internet HellRegistered User regular
    edited December 2010
    Your language is silly.

    Henroid on
    Nobody likes me but that's okay. I'm used to it.
  • Slacker71Slacker71 Registered User regular
    edited December 2010
    Those arn't jailed users as in being punished for infractions. But a user jail keeps processes running on behalf of a user from interfering with another user or the rest of the system.

    Slacker71 on
  • EndEnd Registered User regular
    edited December 2010
    I'm sorry!

    :(

    End on
    I wish that someway, somehow, that I could save every one of us
    zaleiria-by-lexxy-sig.jpgsteam~tinythumb.png
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2010
    Henroid wrote: »
    I found the problem. Somehow our warden server gave two jails the same IP!
    db1# dsh -w web1,web2,web3,web4,web5,web6,web7,web8,web9,db1,db2 jls | grep forums
    web2:      2  172.17.17.133   forums.penny-arcade.com       /usr/jails/forums.penny-arcade.com
    web3:      2  172.17.17.134   forums.penny-arcade.com       /usr/jails/forums.penny-arcade.com
    web4:      1  172.17.17.135   forums.penny-arcade.com       /usr/jails/forums.penny-arcade.com
    web5:      1  172.17.17.132   forums.penny-arcade.com       /usr/jails/forums.penny-arcade.com
    web6:      2  172.17.17.137   forums.penny-arcade.com       /usr/jails/forums.penny-arcade.com
    web8:      4  172.17.17.136   forums.penny-arcade.com       /usr/jails/forums.penny-arcade.com
    web9:      2  172.17.17.133   forums.penny-arcade.com       /usr/jails/forums.penny-arcade.com
    

    Wow. So when those 2 jailed people were posting, were they inadvertently causing problems?

    Different type of jail :wink:

    Those are different web servers that ended up both thinking they were named "172.17.17.133"

    They basically kept yelling at eachother loudly and the other servers got distracted by the commotion.

    Älphämönkëy on
  • EndEnd Registered User regular
    edited December 2010
    alphamonkey's explanation is a lot better than my attempt

    Although I guess he didn't even try to describe what a jail was, and just explained the actual problem. :P

    End on
    I wish that someway, somehow, that I could save every one of us
    zaleiria-by-lexxy-sig.jpgsteam~tinythumb.png
  • HenroidHenroid Radio Demon Internet HellRegistered User regular
    edited December 2010
    Now I feel retarded.

    So is there a perma-fix to preventing this sort of thing, or does it get added to a list of "Hey check this if the database freaks again"?

    Henroid on
    Nobody likes me but that's okay. I'm used to it.
  • EndEnd Registered User regular
    edited December 2010
    That particular issue would probably only happen when you start things up I would think.

    End on
    I wish that someway, somehow, that I could save every one of us
    zaleiria-by-lexxy-sig.jpgsteam~tinythumb.png
  • RamiusRamius Joined: July 19, 2000 Administrator, ClubPA admin
    edited December 2010
    that was not the issue that took down the forums. that was just the cause of the memcache errors after the forums came back up.

    Ramius on
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2010
    Henroid, it's an absurdly specific piece of technical knowledge, you shouldn't feel bad about the confusion.

    End and Slacker have my respect for figuring out what is going on from the snippets of broken alphamonkey-speak.

    End is right, it is rarely going to be an issue. I'm still unsure as to how exactly that caused errors with memcache, but that's something I'll look at when I've had more sleep.

    Älphämönkëy on
  • EndEnd Registered User regular
    edited December 2010
    To be fair, I've dealt with FreeBSD a bit myself, so I'm somewhat familiar. :)

    (And a lot of server stuff in general.)

    End on
    I wish that someway, somehow, that I could save every one of us
    zaleiria-by-lexxy-sig.jpgsteam~tinythumb.png
  • HenroidHenroid Radio Demon Internet HellRegistered User regular
    edited December 2010
    Ramius wrote: »
    that was not the issue that took down the forums. that was just the cause of the memcache errors after the forums came back up.

    I forget if you guys already answered this, but in terms of what took down the forums did you track that issue down yet? Or is this reset (with some minor data loss) an attempt at fixing it without specifically finding it? (which is why we're all here before the forum is actually back up in full swing, I guess)

    Edit - I had a little hiccup just now making this post where it wouldn't load the page (just a perpetual loading bar on the browser from the reply screen). Not sure if that's indicative of much.

    Henroid on
    Nobody likes me but that's okay. I'm used to it.
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2010
    Henroid wrote: »
    Ramius wrote: »
    that was not the issue that took down the forums. that was just the cause of the memcache errors after the forums came back up.

    I forget if you guys already answered this, but in terms of what took down the forums did you track that issue down yet? Or is this reset (with some minor data loss) an attempt at fixing it without specifically finding it? (which is why we're all here before the forum is actually back up in full swing, I guess)

    I haven't posted about it yet, that will be the last thing I do tonight :wink:

    Älphämönkëy on
  • PaperLuigi44PaperLuigi44 My amazement is at maximum capacity. Registered User regular
    edited December 2010
    I just want to say thanks alpha (and anyone who helped you out) for everything, now I no longer have to worry about forum withdrawal <3

    PaperLuigi44 on
    - Fixed a bug where the Moon was upside down.
    - Fixed a weird door.
  • TiamatZTiamatZ Ghost puns The Banette of my existenceRegistered User regular
    edited December 2010
    I don't know if this was posted already or not, but seems anything after Dec 1 was lost/deleted.

    Anywhoo, I'd like to thank alpha for his dedication on bringing back the forums.

    Also, any new details regarding the new forum change for Feb yet?

    TiamatZ on
  • PaperLuigi44PaperLuigi44 My amazement is at maximum capacity. Registered User regular
    edited December 2010
    TiamatZ wrote: »
    I don't know if this was posted already or not, but seems anything after Dec 1 was lost/deleted.

    Actually I just noticed that myself, as I lost the minor edits I made to this thread (just a thread title change and embedding a video, no big loss).

    PaperLuigi44 on
    - Fixed a bug where the Moon was upside down.
    - Fixed a weird door.
  • unintentionalunintentional smelly Registered User regular
    edited December 2010
    clicking the "go to first unread post" button in the MSPA thread is just opening the latest page, and not redirecting me to a new post

    and when I scroll there isn't a post

    maybe this just means that the guy deleted his post, but I thought I'd mention it

    unintentional on
  • HenroidHenroid Radio Demon Internet HellRegistered User regular
    edited December 2010
    Did you guys intentionally turn the forum back on? Because the regular URL works now.

    Edit - Oh goddamn it I keep forgetting I'm not on the west coast NEVERMIND.

    Henroid on
    Nobody likes me but that's okay. I'm used to it.
  • VALVEjunkieVALVEjunkie Registered User regular
    edited December 2010
    thanks for handling this situation swiftly alpha!(and all others involved) :)

    VALVEjunkie on
  • SoundsPlushSoundsPlush yup, back. Registered User regular
    edited December 2010
    I just want to say thanks alpha (and anyone who helped you out) for everything, now I no longer have to worry about forum withdrawal <3

    Yay!

    SoundsPlush on
    s7Imn5J.png
  • Eat it You Nasty Pig.Eat it You Nasty Pig. tell homeland security 'we are the bomb'Registered User regular
    edited December 2010
    thanks for busting ass to get the forums back up alpha, I was getting the shakes

    Eat it You Nasty Pig. on
    NREqxl5.jpg
  • HenroidHenroid Radio Demon Internet HellRegistered User regular
    edited December 2010
    Where's the love for Ramius?

    Henroid on
    Nobody likes me but that's okay. I'm used to it.
  • Undead ScottsmanUndead Scottsman Registered User regular
    edited December 2010
    Just want to say thank you to everyone involved for burning the midnight oil to get the forums back.

    Undead Scottsman on
  • proXimityproXimity Registered User regular
    edited December 2010
    So is there anything special about the "secret" forums?

    proXimity on
    camo_sig2.png
  • admanbadmanb unionize your workplace Seattle, WARegistered User regular
    edited December 2010
    proXimity wrote: »
    So is there anything special about the "secret" forums?

    From our perspective, no.

    admanb on
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2010
    I am intentionally restarting the MySQL server now to update the configuration values.

    Should be super short.

    EDIT: All done! Okay, that's the last of my work tonight.

    Älphämönkëy on
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2010
    admanb wrote: »
    proXimity wrote: »
    So is there anything special about the "secret" forums?

    From our perspective, no.

    I used the secret forums to test out the forums with a smaller group of people before opening it up to the community at large. I went into IRC and basically handed them a URL and said "go nuts"

    The secret forums are now disabled :wink:

    Älphämönkëy on
  • EchoEcho Moderator mod
    edited December 2010
    Can't say I'm good at BSD, but does that mean the forums are distributed across seven servers?

    It's always interesting when alpha talks nerdy to us. Rawr.

    Echo on
    Echo wrote: »
    Let they who have not posted about their balls in the wrong thread cast the first stone.
  • EvilBadmanEvilBadman DO NOT TRUST THIS MAN Registered User regular
    edited December 2010
    Thanks Alpha!

    EvilBadman on
    FyreWulff wrote: »
    I should note that Badman is fucking awesome
    XBL- Evil Badman; Steam- EvilBadman; Twitter - EvilBadman
  • Mr_RoseMr_Rose 83 Blue Ridge Protects the Holy Registered User regular
    edited December 2010
    Echo wrote: »
    Can't say I'm good at BSD, but does that mean the forums are distributed across seven servers?

    It's always interesting when alpha talks nerdy to us. Rawr.
    Either seven servers or seven virtual instances of some sort that might or might not be on different actual boxes. I too am always intereted in the technicals of running such a large forum.

    Anyway, first; yay forums. Many props for getting things back up so fast.
    second; we apparently did lose some threads in CF too, like the new 40K thread. Don't think anything significant is missing though.

    Mr_Rose on
    ...because dragons are AWESOME! That's why.
    Nintendo Network ID: AzraelRose
    DropBox invite link - get 500MB extra free.
  • AriviaArivia Registered User
    edited December 2010
    The smilie box is all screwed up. (safari on OS X 10.6)

    Arivia on
    huntresssig.jpg
  • ÄlphämönkëyÄlphämönkëy Registered User regular
    edited December 2010
    Mr_Rose wrote: »
    Echo wrote: »
    Can't say I'm good at BSD, but does that mean the forums are distributed across seven servers?

    It's always interesting when alpha talks nerdy to us. Rawr.
    Either seven servers or seven virtual instances of some sort that might or might not be on different actual boxes. I too am always intereted in the technicals of running such a large forum.

    Anyway, first; yay forums. Many props for getting things back up so fast.
    second; we apparently did lose some threads in CF too, like the new 40K thread. Don't think anything significant is missing though.

    I updated the OP with details of what happened.

    I'm tired, so I'm going to briefly describe the jail system. If you have any follow up questions, I'll answer them tomorrow in more detail.

    In short the forums can run on up to seven physical machines. Those machines are shared with other sites that we run (like the PA mainsite). I use read-only jails as a way of isolating each site that we host here at Penny Arcade and creating a reliable and reproducible environment. This also has an added benefit of being fairly secure. Assuming someone is able to find a security flaw in vBulletin, they will end up with unprivileged access to a read-only jail. Of course they could screw up the database, but the other sites would not be impacted (each site has it's own database server instance & VLANs keep the networks physically separate).

    Älphämönkëy on
  • AegeriAegeri Plateau of LengRegistered User regular
    edited December 2010
    Thanks so much for all your hard work Alpha and everyone else as well who helped.

    Aegeri on
    The Roleplayer's Guild: My blog for roleplaying games, advice and adventuring.
  • THEPAIN73THEPAIN73 Shiny. Real shiny.Registered User regular
    edited December 2010
    Just want to say thank you to everyone involved for burning the midnight oil to get the forums back.

    Ditto. Came in here to say just this.

    THEPAIN73 on
    Facebook | Amazon | Twitter | Youtube | PSN: ThePain73 | Steam: ThePain73
    3DS FC: 5343-7720-0490
  • L Ron HowardL Ron Howard Registered User regular
    edited December 2010
    Thanks alpha and Ramius and all!

    L Ron Howard on
  • streeverstreever Registered User regular
    edited December 2010
    Thank you, all.

    I'm not a HUGE mysql guy, so I may be wrong, but I had a feeling of horror when you described killing the mysql process ;-) If mysql is running and has users and you kill it's process, it will corrupt the tables, pretty much every time. That will lead to crashing & burning.

    You can issue a stop or restart command quite safely however.

    Apologies if I misunderstood what you posted: just trying to be helpful! Mysql isn't really that bad, but it does need to have repair operations run occasionally and shouldn't be killed if it can be avoided.

    streever on
  • MichaelLCMichaelLC In what furnace was thy brain? ChicagoRegistered User regular
    edited December 2010
    Thanks guys!

    I was very lost without the forums. I nearly started a conversation with a real person on the train.

    MichaelLC on
    "Never believe management about anything anywhere." -Aistan
  • MKRMKR Registered User regular
    edited December 2010
    streever wrote: »
    Thank you, all.

    I'm not a HUGE mysql guy, so I may be wrong, but I had a feeling of horror when you described killing the mysql process ;-) If mysql is running and has users and you kill it's process, it will corrupt the tables, pretty much every time. That will lead to crashing & burning.

    You can issue a stop or restart command quite safely however.

    Apologies if I misunderstood what you posted: just trying to be helpful! Mysql isn't really that bad, but it does need to have repair operations run occasionally and shouldn't be killed if it can be avoided.

    I forget what he said, but there was some problem with just stopping and starting or restarting.

    Also, here are more red pandas: http://www.zooborns.typepad.com/zooborns/red_panda/

    MKR on
  • TomantaTomanta Registered User regular
    edited December 2010
    Thanks for all the hard work Alpha, Ramius, everyone else. Yesterday at work without the forums was extremely boring, even if it did let me spend longer than necessary figuring out why I couldn't get samba working at home.

    Tomanta on
  • Goose!Goose! That's me, honey Show me the way home, honeyRegistered User regular
    edited December 2010
    Yes, thanks very much to Alpha and Ramius.

    My only regret is that my only awesome post ever was lost in the fire.

    Goose! on
Sign In or Register to comment.