Wow. So when those 2 jailed people were posting, were they inadvertently causing problems?
Heh.
Naw, jails is a FreeBSD thing. It's like a lightweight partial virtualization (similar but not quite like VMWare, Parallels, etc), but it only works within the context of FreeBSD (the OS that runs the forums). I'm not sure that was that great of an explanation, but at worst: it's a server thing with making one server look more like multiple.
End on
I wish that someway, somehow, that I could save every one of us
0
HenroidMexican kicked from Immigration ThreadCentrism is Racism :3Registered Userregular
Those arn't jailed users as in being punished for infractions. But a user jail keeps processes running on behalf of a user from interfering with another user or the rest of the system.
Henroid, it's an absurdly specific piece of technical knowledge, you shouldn't feel bad about the confusion.
End and Slacker have my respect for figuring out what is going on from the snippets of broken alphamonkey-speak.
End is right, it is rarely going to be an issue. I'm still unsure as to how exactly that caused errors with memcache, but that's something I'll look at when I've had more sleep.
that was not the issue that took down the forums. that was just the cause of the memcache errors after the forums came back up.
I forget if you guys already answered this, but in terms of what took down the forums did you track that issue down yet? Or is this reset (with some minor data loss) an attempt at fixing it without specifically finding it? (which is why we're all here before the forum is actually back up in full swing, I guess)
Edit - I had a little hiccup just now making this post where it wouldn't load the page (just a perpetual loading bar on the browser from the reply screen). Not sure if that's indicative of much.
that was not the issue that took down the forums. that was just the cause of the memcache errors after the forums came back up.
I forget if you guys already answered this, but in terms of what took down the forums did you track that issue down yet? Or is this reset (with some minor data loss) an attempt at fixing it without specifically finding it? (which is why we're all here before the forum is actually back up in full swing, I guess)
I haven't posted about it yet, that will be the last thing I do tonight
Älphämönkëy on
0
PaperLuigi44My amazement is at maximum capacity.Registered Userregular
edited December 2010
I just want to say thanks alpha (and anyone who helped you out) for everything, now I no longer have to worry about forum withdrawal
PaperLuigi44 on
0
TiamatZGhost punsThe Banette of my existenceRegistered Userregular
edited December 2010
I don't know if this was posted already or not, but seems anything after Dec 1 was lost/deleted.
Anywhoo, I'd like to thank alpha for his dedication on bringing back the forums.
Also, any new details regarding the new forum change for Feb yet?
TiamatZ on
0
PaperLuigi44My amazement is at maximum capacity.Registered Userregular
So is there anything special about the "secret" forums?
From our perspective, no.
I used the secret forums to test out the forums with a smaller group of people before opening it up to the community at large. I went into IRC and basically handed them a URL and said "go nuts"
Can't say I'm good at BSD, but does that mean the forums are distributed across seven servers?
It's always interesting when alpha talks nerdy to us. Rawr.
Either seven servers or seven virtual instances of some sort that might or might not be on different actual boxes. I too am always intereted in the technicals of running such a large forum.
Anyway, first; yay forums. Many props for getting things back up so fast.
second; we apparently did lose some threads in CF too, like the new 40K thread. Don't think anything significant is missing though.
Can't say I'm good at BSD, but does that mean the forums are distributed across seven servers?
It's always interesting when alpha talks nerdy to us. Rawr.
Either seven servers or seven virtual instances of some sort that might or might not be on different actual boxes. I too am always intereted in the technicals of running such a large forum.
Anyway, first; yay forums. Many props for getting things back up so fast.
second; we apparently did lose some threads in CF too, like the new 40K thread. Don't think anything significant is missing though.
I updated the OP with details of what happened.
I'm tired, so I'm going to briefly describe the jail system. If you have any follow up questions, I'll answer them tomorrow in more detail.
In short the forums can run on up to seven physical machines. Those machines are shared with other sites that we run (like the PA mainsite). I use read-only jails as a way of isolating each site that we host here at Penny Arcade and creating a reliable and reproducible environment. This also has an added benefit of being fairly secure. Assuming someone is able to find a security flaw in vBulletin, they will end up with unprivileged access to a read-only jail. Of course they could screw up the database, but the other sites would not be impacted (each site has it's own database server instance & VLANs keep the networks physically separate).
Älphämönkëy on
0
AegeriTiny wee bacteriumsPlateau of LengRegistered Userregular
edited December 2010
Thanks so much for all your hard work Alpha and everyone else as well who helped.
I'm not a HUGE mysql guy, so I may be wrong, but I had a feeling of horror when you described killing the mysql process ;-) If mysql is running and has users and you kill it's process, it will corrupt the tables, pretty much every time. That will lead to crashing & burning.
You can issue a stop or restart command quite safely however.
Apologies if I misunderstood what you posted: just trying to be helpful! Mysql isn't really that bad, but it does need to have repair operations run occasionally and shouldn't be killed if it can be avoided.
streever on
0
MichaelLCIn what furnace was thy brain?ChicagoRegistered Userregular
edited December 2010
Thanks guys!
I was very lost without the forums. I nearly started a conversation with a real person on the train.
I'm not a HUGE mysql guy, so I may be wrong, but I had a feeling of horror when you described killing the mysql process ;-) If mysql is running and has users and you kill it's process, it will corrupt the tables, pretty much every time. That will lead to crashing & burning.
You can issue a stop or restart command quite safely however.
Apologies if I misunderstood what you posted: just trying to be helpful! Mysql isn't really that bad, but it does need to have repair operations run occasionally and shouldn't be killed if it can be avoided.
I forget what he said, but there was some problem with just stopping and starting or restarting.
Thanks for all the hard work Alpha, Ramius, everyone else. Yesterday at work without the forums was extremely boring, even if it did let me spend longer than necessary figuring out why I couldn't get samba working at home.
Tomanta on
0
Goose!That's me, honeyShow me the way home, honeyRegistered Userregular
edited December 2010
Yes, thanks very much to Alpha and Ramius.
My only regret is that my only awesome post ever was lost in the fire.
Posts
Heh.
Naw, jails is a FreeBSD thing. It's like a lightweight partial virtualization (similar but not quite like VMWare, Parallels, etc), but it only works within the context of FreeBSD (the OS that runs the forums). I'm not sure that was that great of an explanation, but at worst: it's a server thing with making one server look more like multiple.
Different type of jail
Those are different web servers that ended up both thinking they were named "172.17.17.133"
They basically kept yelling at eachother loudly and the other servers got distracted by the commotion.
Although I guess he didn't even try to describe what a jail was, and just explained the actual problem. :P
So is there a perma-fix to preventing this sort of thing, or does it get added to a list of "Hey check this if the database freaks again"?
End and Slacker have my respect for figuring out what is going on from the snippets of broken alphamonkey-speak.
End is right, it is rarely going to be an issue. I'm still unsure as to how exactly that caused errors with memcache, but that's something I'll look at when I've had more sleep.
(And a lot of server stuff in general.)
I forget if you guys already answered this, but in terms of what took down the forums did you track that issue down yet? Or is this reset (with some minor data loss) an attempt at fixing it without specifically finding it? (which is why we're all here before the forum is actually back up in full swing, I guess)
Edit - I had a little hiccup just now making this post where it wouldn't load the page (just a perpetual loading bar on the browser from the reply screen). Not sure if that's indicative of much.
I haven't posted about it yet, that will be the last thing I do tonight
Anywhoo, I'd like to thank alpha for his dedication on bringing back the forums.
Also, any new details regarding the new forum change for Feb yet?
Actually I just noticed that myself, as I lost the minor edits I made to this thread (just a thread title change and embedding a video, no big loss).
and when I scroll there isn't a post
maybe this just means that the guy deleted his post, but I thought I'd mention it
Edit - Oh goddamn it I keep forgetting I'm not on the west coast NEVERMIND.
Yay!
Pluto was a planet and I'll never forget
From our perspective, no.
Should be super short.
EDIT: All done! Okay, that's the last of my work tonight.
I used the secret forums to test out the forums with a smaller group of people before opening it up to the community at large. I went into IRC and basically handed them a URL and said "go nuts"
The secret forums are now disabled
It's always interesting when alpha talks nerdy to us. Rawr.
Anyway, first; yay forums. Many props for getting things back up so fast.
second; we apparently did lose some threads in CF too, like the new 40K thread. Don't think anything significant is missing though.
Nintendo Network ID: AzraelRose
DropBox invite link - get 500MB extra free.
I updated the OP with details of what happened.
I'm tired, so I'm going to briefly describe the jail system. If you have any follow up questions, I'll answer them tomorrow in more detail.
In short the forums can run on up to seven physical machines. Those machines are shared with other sites that we run (like the PA mainsite). I use read-only jails as a way of isolating each site that we host here at Penny Arcade and creating a reliable and reproducible environment. This also has an added benefit of being fairly secure. Assuming someone is able to find a security flaw in vBulletin, they will end up with unprivileged access to a read-only jail. Of course they could screw up the database, but the other sites would not be impacted (each site has it's own database server instance & VLANs keep the networks physically separate).
Ditto. Came in here to say just this.
3DS FC: 5343-7720-0490
I'm not a HUGE mysql guy, so I may be wrong, but I had a feeling of horror when you described killing the mysql process ;-) If mysql is running and has users and you kill it's process, it will corrupt the tables, pretty much every time. That will lead to crashing & burning.
You can issue a stop or restart command quite safely however.
Apologies if I misunderstood what you posted: just trying to be helpful! Mysql isn't really that bad, but it does need to have repair operations run occasionally and shouldn't be killed if it can be avoided.
I was very lost without the forums. I nearly started a conversation with a real person on the train.
I forget what he said, but there was some problem with just stopping and starting or restarting.
Also, here are more red pandas: http://www.zooborns.typepad.com/zooborns/red_panda/
My only regret is that my only awesome post ever was lost in the fire.