Where did Alpha Go? (read this)

Älphämönkëy

Hey all.
The forums have been picking up in usage recently, and I've been hiding in the background working on keeping all the features afloat. Turns out I can't. phpBB just plain doesn't scale for searching without adding hardware or doing massive code rewrites. So searching is going to be removed from our phpBB powered forum permanently. Before the lync mob comes to my appartment, atleast hear my second announcement. I finally got the new forum code base in a place where I like it. Cheifly, someone else did a lot of the hardwork. I need to make a few more phone calls, but I think the new forum software will be a rails app by the name of Opinion.
Let me explain why I chose opinion. Tobi and the guys over at leetsoft, are very talented developers. Infact, these guys write the code that was used as a framework for the penny arcade main page. This also provides people with an outlet, as if you want to see a feature on the forums, write a patch. If it doesn't suck, I'll throw it in. Opinion isn't perfect, I have to roll in a lot of changes to their source code. Likewise, Opinion's default template was designed to be an "interactive blog" and we definatly aren't an interactive blog. So that will need to change. But the core of it is a good app, that will scale really well, and will be more responsive in general.

I'm going to try and get a demo application hosted so you guys can see what it looks like and beat on it a bit.

Okay, now you can send the lynch mob to my appartment.
-Alpha

TDL

Oh man, how am I going to ego search now?

But seriously, I'm sure we'll all get over it.

Orikaeshigitae

Yay, stability forthcoming!

Alpha.

Echo

Rails

redx

duh... forum not break so much. good.

me happy.

no understad rails.

love alpha.


:edit: hey alpha got off his ass and is finaly inserting word matches again! Yay editing abounds.

Thanatos

Hmmmm... Nonworking forum with searches, or working forum without searches... tough call. :P

You kick ass, Alpha.

contraband

redx wrote:

duh... forum not break so much. good.

me happy.

no understad rails.

love alpha.

you know we would all be on our asses over at gamefaqs or something if it wasn't for you

Toast

:^:

Rear Admiral Choco

Hell, I barely use the search function anyway.

Thanks. :^:

elecguts

The absence of a search function may hurt Linksville a bit, although people still posted old links even with the search function. It just might happen tad more often now.

redx

so... mods can still search, right?

People can't get away with doing naughy things without being found out.

Bionic Monkey

Burn Him!!


Seriously though, thanks for all the hard work Alpha, and uh... guys at Leetsoft.

Henroid

I never used the search function anyhow.

I think it is a good call. Someone, somewhere, is bound to know where a thread is floating if a duplicate is made.

Additionally, maybe I should start up the Game On Reference Desk again.

Thanks for the work you do Alpha.

Edit - I went ahead and made it. I'd like to think this is for all forumers, not just G&T (it's there because it fits).

Deusfaux

what are some of the other major forums around the net using? how do they get on?

edit: er, and how long between disabling search (and getting it back via the new code?)

Pheezer

Deusfaux wrote:

what are some of the other major forums around the net using? how do they get on?

edit: er, and how long between disabling search (and getting it back via the new code?)

I think the more common answer to load issues is to throw more and more hardware at the problem.

The forums are a pretty small segment of PA's readership though, and accordingly, don't really warrant the whole "invest whatever is necessary to keep the current codebase living" approach.

The largest webforum around, based on http://rankings.big-boards.com/ has 3400867 members in total, but gets by with phpBB 2.

IGN's forums software is indeterminate, at least to me, but according to the credits they're paying someone to run it off of an Oracle database for them.

Most of the really big forums derive significant revenue from their forums. PA is pretty big for a forum that probably does not recoup its own costs directly (ignoring ad revenue stemming from traffic that would visit the front page only because that given user visits the forums because that is not realistic).

Deusfaux

haha i wonder who wrote the little blurbs for us:

Hardware: A shared web cluster powers the webserving front end to the site, and is matched by a small, private MySQL 4 database server on the backend.

Software: Running a very hacked version of phpBB. Most of the code has been optimized by hand, and several mods were developed solely for use with the penny arcade forums. Additionally, the developers have come up with several security enhancements which they have re-released to the public.

Number of moderators: 26



edit: that site has neat stats. while we're only around 170th for posts (seems kind of a silly thing to rank by first and foremost) and we drop to like 600 for members, we shoot way up for traffic (40-some) and daily page views - though those are probably related to the front page somehow?

we have a high average online users count too at 22nd with (500)

so many lurkers!!!

Pheezer

Deusfaux wrote:

haha i wonder who wrote the little blurbs for us:

Hardware: A shared web cluster powers the webserving front end to the site, and is matched by a small, private MySQL 4 database server on the backend.

Software: Running a very hacked version of phpBB. Most of the code has been optimized by hand, and several mods were developed solely for use with the penny arcade forums. Additionally, the developers have come up with several security enhancements which they have re-released to the public.

Number of moderators: 26



edit: that site has neat stats. while we're only around 170th for posts (seems kind of a silly thing to rank by first and foremost) and we drop to like 600 for members, we shoot way up for traffic (40-some) and daily page views - though those are probably related to the front page somehow?

we have a high average online users count too at 22nd with (500)

so many lurkers!!!

I'm pretty sure they're seperating the non-forum hits from the forum hits. Actually I'm absolutely sure. PA is a huge, huge site.

That members stat is the telling part though. You can expect ad revenue to be related most directly to the number of members, meaning that on average, we make as much in ad revenue as forums with around 600 users. But there's no way in hell any other forum with 600 or so users puts as much load on their server hardware as we do.

Deusfaux

for anyone whom it wasnt also immediately clear: our auto pruning kills a fair amount of our posts

certainly more than back in the day when you could go back like through 30 pages of material or more.

so these other forums with piles of servers can probably afford to house massive amounts of history better than we could (and rank higher according to dimensions that use postcount)

I think Alpha still has the on-going stats for total forum posts and things though, maybe he could tell us for curiousity's sake.

Just_Bri_Thanks

lurks out

Pheezer

Deusfaux wrote:

for anyone whom it wasnt also immediately clear: our auto pruning kills a fair amount of our posts

certainly more than back in the day when you could go back like through 30 pages of material or more.

so these other forums with piles of servers can probably afford to house massive amounts of history better than we could (and rank higher according to dimensions that use postcount)

I think Alpha still has the on-going stats for total forum posts and things though, maybe he could tell us for curiousity's sake.

This is a good point. As of this moment there are only approximately 559,000 posts in total in the database. I don't think the current pruning schema allows it to really get much higher.

I don't know if alpha keeps "lifetime" stats, though.

Deusfaux

it was him or Ramius who once spouted off a few stats that as far as I remember, included stuff from before the big wipe, even...

contraband

I was curious, so now I know that the japanese 2channel is apparently the largest forum in the world

scarlet st.

Alpha, I will take you out to lunch someday

:^: Minnesota

mcc

Thanks for the update.

Will the Opinion forum support searching once it is ready?

Henroid

Hey, uh, this kinda just occured to me, but seeing as adminship has changed to being Alpha and Whippy, will the Monkey Den be renamed to reflect this?

Didn't think this deserved it's own thread.

Dog

No.

Henroid

SuprSaiyanGoku47 wrote:

No.

But it isn't Alpha Monkey and denihilist!

Legacy

Henroid wrote:

SuprSaiyanGoku47 wrote:

No.

But it isn't Alpha Monkey and denihilist!

It still works.

Dog

The topic has come up a few times already, Whippy said the name will stay.

jothki

As long as everything works more or less in the same way on the user side, I'll be happy. I love the current feel of the forums.

What will we be seeing when the shift happens?

Henroid

SuprSaiyanGoku47 wrote:

The topic has come up a few times already, Whippy said the name will stay.

Ah okay. Just wonderin'. :P

Whippy

Basically I just can't think of anything witty.

:-p

Älphämönkëy

pheezer FD wrote:

Deusfaux wrote:

what are some of the other major forums around the net using? how do they get on?

edit: er, and how long between disabling search (and getting it back via the new code?)

I think the more common answer to load issues is to throw more and more hardware at the problem.

Yup. The expensive part of a forum isn't the PHP part (once you optimize things) it is the database part. So most people introduce some form of replication. Extra hardware is where we would be able to save ourselves, but I would rather see extra hardware spent on things that would help take stress off the mainsite, maybe some dedicated power for serving ads quicker.

The forums are a pretty small segment of PA's readership though, and accordingly, don't really warrant the whole "invest whatever is necessary to keep the current codebase living" approach.

Yup. Robert has offered to buy us software and even a bit of hardware if it would make things easier, but again, my priority in terms of speed is the mainsite.

The largest webforum around, based on http://rankings.big-boards.com/ has 3400867 members in total, but gets by with phpBB 2.

That's gaiaonline. To say that they use phpBB2 is somewhat of an exaggeration. They started with phpBB2, and basically rewrote every SQL query. They also use a server farm to host their stuff. They have replication of databases like you wouldn't believe and a host of linux webservers up front. Last time I talked with Lanzer they basically had an entire set of racks devoted to them. A lot of the optimizations we use he developed, and I'm fairly confident he is using a few of my hacks as well.

IGN's forums software is indeterminate, at least to me, but according to the credits they're paying someone to run it off of an Oracle database for them.

Yeah. If I had a dedicated Oracle 9i enterprise server and a full-time DBA, you wouldn't see any database issues. MySQL corrupting is the problem, and row level locks in a database is the solution. The problem is, MySQL sucks with row level locks in 4, and MySQL 5 seems like a little kid that is now wearing Pull-Ups. Yeah, you don't shit yourself as much anymore, but when you do, its a mess to clean up. PostgreSQL will be our saving grace in the next iteration of the forums, and it will use some cool features for accomplishing things like permissions and such. But, that is a story for another time...

Most of the really big forums derive significant revenue from their forums. PA is pretty big for a forum that probably does not recoup its own costs directly (ignoring ad revenue stemming from traffic that would visit the front page only because that given user visits the forums because that is not realistic).

Exactly. We are fairly unique in that aspect. I mean, we don't even have the google adsense ads anymore, so the forums provide no revenue for Penny Arcade.

Älphämönkëy

Deusfaux wrote:

haha i wonder who wrote the little blurbs for us:

Hardware: A shared web cluster powers the webserving front end to the site, and is matched by a small, private MySQL 4 database server on the backend.

Software: Running a very hacked version of phpBB. Most of the code has been optimized by hand, and several mods were developed solely for use with the penny arcade forums. Additionally, the developers have come up with several security enhancements which they have re-released to the public.

Number of moderators: 26



edit: that site has neat stats. while we're only around 170th for posts (seems kind of a silly thing to rank by first and foremost) and we drop to like 600 for members, we shoot way up for traffic (40-some) and daily page views - though those are probably related to the front page somehow?

we have a high average online users count too at 22nd with (500)

so many lurkers!!!

Heh. That was me a long time ago. Back in the HomeLAN era actually. Those big-boards stats are a bit messed up as I turned on some protection code that prevents spiders from crawling the wrong info. Problem is, that prevents them from crawling as well. So we are 600th in members using really old data. I bet if I contacted the guys at big-boards, I could get them access to a raw numbers page, and we would shoot up considerably. I think our posts are also semi-protected. I don't know, I don't keep track of these things very well.

Älphämönkëy

mcc wrote:

Thanks for the update.

Will the Opinion forum support searching once it is ready?

Yup. We will be able to have a lot more "fancy" effects since we will be freeing up the DB from constantly being beaten on. For those who enjoy buzzwords, we will be using fun AJAXy bits to make it feel more like an application.

Älphämönkëy

scarlet st. wrote:

Alpha, I will take you out to lunch someday

:^: Minnesota

I appreciate the offer, but I'm only in Minnesota in the summers, during the school year I hide elsewhere. Even so, outside of PAX, I don't meet up with many people. The one exception being Snowcone, I met him and his wife in Texas a few years back.

Älphämönkëy

Deusfaux wrote:

it was him or Ramius who once spouted off a few stats that as far as I remember, included stuff from before the big wipe, even...

That was me. I can't get at those stats any more due to pruning. I can give you some estimates though. This post here will be #22655487 in thread #216077. Those numbers started at 1 when we relaunched.

Älphämönkëy

jothki wrote:

As long as everything works more or less in the same way on the user side, I'll be happy. I love the current feel of the forums.

What will we be seeing when the shift happens?

The forums will look and feel a bit different. I would like to port BBCode over to opinion, but I don't know if I will have the time. I might make a list of code bounties, for kind developers to tackle. If so, that would be high up there.

Finster

I've been wondering this since a few days ago when I landed a Web programming (PHP) job with a local company here. They do colocation and all that other stuff as well. The whole place runs on FreeBSD/PHP/MySQL.

So, I was wondering what (basic) configuration is used to host the forums. My understanding is that it is a single server using some of the software noted in Alphamonkey's sig, but I guess I'm more curious about what actually goes into setting up the server as far as backend stuff.

I think there was a thread on this very subject many moons ago, but I have no idea where that would be now.

Älphämönkëy

Finster wrote:

I've been wondering this since a few days ago when I landed a Web programming (PHP) job with a local company here. They do colocation and all that other stuff as well. The whole place runs on FreeBSD/PHP/MySQL.

So, I was wondering what (basic) configuration is used to host the forums. My understanding is that it is a single server using some of the software noted in Alphamonkey's sig, but I guess I'm more curious about what actually goes into setting up the server as far as backend stuff.

I think there was a thread on this very subject many moons ago, but I have no idea where that would be now.

Normally I'd get into it. But I don't have the time. I'd say you could search for it, but... well... yeah.

FyreWulff

I'm looking forward to a forumized version of this software to use on my site.. that is, unless I get the money for vBB first.

My patience has run out with the phpBB developers.


I imagine the jail system will come over, or will that be forgotten? (or added later on)