I Ruby on Rails, and have built a small local app for my organization with it before. It was wonderful to work in. I'd love to get my hands on this and help give back to the P-A community.
Inkoate on
The man in black fled across the desert, and the gunslinger followed.
It is and it isn't. It is a nice app, has a lot of features and such, but it only supports MySQL and it doesn't support FastCGI. I could port their code to work with FastCGI, but I'll argue that isn't my job. So yeah, on the whole, nice app, for us, not so much.
Is our volume really that much higher than so many other companies that use it that we can no longer be contained in SQL?
It seems some really major forums skate by on SQL.
it's not SQL itself, it's mySQL.
really big internet forums are usually not running off mySQL. there are exceptions. i've seen a huge forum once that was running an Oracle database, which I believe that if money was no object, would be the SQL database Alpha would like to run these forums off of.
Is our volume really that much higher than so many other companies that use it that we can no longer be contained in SQL?
It seems some really major forums skate by on SQL.
I assume you're talking about MySQL (the software) and not SQL (the language). Indeed, a lot of huge sites use MySQL (Slashdot, Wikipedia, I'm assuming the aforementioned biggest forum on big-boards.com too), but they have tons of servers to distribute the load between. Meanwhile, last I checked these forums didn't even use a dedicated database server - it's all on one machine. It's amazing, really.
Now let's look at the way phpBB deals with searching. Every time a new post is made, it's split up into separate words and each word (except for really short ones and a bunch of very frequently used one, like "think", "fucking" and "cocks") is inserted into one big table. The purpose of this is to have an index of every single searchable word and then use it to speed up search queries.
For example, let's say the search word table looks like this:
id word
1 spore
2 katamari
3 damacy
4 metal
5 gear
6 solid
...
Then someone decides to make a new thread dedicated to Spore and posts "Hay guys, did you hear about this game Spore by Will Wright? It's going to be awesome." phpBB splits the post into words and first discards those that are 3 letters long or less. Then it discards the really popular ones that don't really help searching. So now we're left with "spore", "wright" and "awesome". phpBB then checks to see if every single one of those words is already in the big word table. Let's say "spore" is there, but "wright" and "awesome" are not. Now we need to add them:
id word
1 spore
2 katamari
3 damacy
4 metal
5 gear
6 solid
...
121446 wright
121447 awesome
Finally, we have another table to match words with certain posts. Let's say the above Spore post was #12345. So phpBB connects the three words with this post:
post word
...
12345 1
12345 121446
12345 121447
So when somebody makes a search for "will wright spore", phpBB quickly matches the words "wright" and "spore" (through their IDs) with post #12345 and outputs the results.
Now imagine how much load a single machine is under while trying to deal with multiple posts per second like that. Remember "could not insert word matches"? That's MySQL throwing in the towel. What phpBB does might have been a suitable solution for small-to-medium sized forums back before MySQL implemented full-text search, but it's clearly too inefficient for PA. Besides, like alpha said, MySQL simply lacks certain features that would have helped avoid table corruption.
The moral of the story is we better get ready to welcome our new Postgre overlords.
Well, I'm going to throw in one more request stolen from Librarium...
A check mark next to threads that we have posted in or are following. G&T moves hella fast and there's only a few threads that I'm interested in posting/viewing.
I realize we have the "Watch this topic for replies" option, but does that even work?
I used to use the egosearch to keep track of stuff, though it doesn't track threads that we are interested in but haven't posted in yet.
Edit: Actually, would it be possible to keep the user searching active while disabling the word tracking until you get the new setup running?
I'm wondering why the the option of asking the membership for financial help in acquiring new and more servers was never put forth? I mean new software optimizations are cool and all, but there's really only so much you can do with that before you need new machines.
There's 42k members on the member list, and while most inactive there's probably a solid number of lurkers as well who might help. Further I think if the financial situation was put in front of people, it would really make them think about how much we use this forum and might cut back on redundant or retarded threads. I mean if 10k people all gave about $10 in donations per year, that's a nice $100k. I'm willing to break into my piggy bank if it means getting standard features, and increased forum stability:
I'm wondering why the the option of asking the membership for financial help in acquiring new and more servers was never put forth? I mean new software optimizations are cool and all, but there's really only so much you can do with that before you need new machines.
There's 42k members on the member list, and while most inactive there's probably a solid number of lurkers as well who might help. Further I think if the financial situation was put in front of people, it would really make them think about how much we use this forum and might cut back on redundant or retarded threads. I mean if 10k people all gave about $10 in donations per year, that's a nice $100k. I'm willing to break into my piggy bank if it means getting standard features, and increased forum stability:
Realistically, you have to count on about a 5% response rate from those capable of responding to such a plea. "Capable" meaning "over 18."
I think you're looking at more like $1000, if that.
I'm wondering why the the option of asking the membership for financial help in acquiring new and more servers was never put forth?
If I remember right, it's because we didn't want to place the onus of fixing the forums on the forumers.
I am pretty sure the problem would be someone acting up and bitching about getting banned even though they paid for the privilege of posting. Ya know, the kind of shit that TTB puts up with.
Slungsolow on
fuck your forums, fuck your administrator and fuck dynagrip for getting away with the long troll.
Munkus BeaverYou don't have to attend every argument you are invited to.Philosophy: Stoicism. Politics: Democratic SocialistRegistered User, ClubPAregular
edited March 2006
Hush. I liked it.
Munkus Beaver on
Humor can be dissected as a frog can, but dies in the process.
I'm all for more stability, though I'm wondering if we're all going to have to register new accounts? I imagine everyone trying to re-register could create quite a bit of strain, but I dont have the tech knowledge to know if thats a concern or not. I also presume that all the existing forum threads and so on will get wiped?
It would also be kind of cool to have a post count leak on the last day of this iteration of the forums.
I wouldn't mind reregistering. If only because I've wanted to change my name for a while, but it's already taken. Which, normally I would respect, but it seems to have been taken by someone that either never posts, or has long since left the forums.
Is our volume really that much higher than so many other companies that use it that we can no longer be contained in SQL?
It seems some really major forums skate by on SQL.
I assume you're talking about MySQL (the software) and not SQL (the language). Indeed, a lot of huge sites use MySQL (Slashdot, Wikipedia, I'm assuming the aforementioned biggest forum on big-boards.com too), but they have tons of servers to distribute the load between. Meanwhile, last I checked these forums didn't even use a dedicated database server - it's all on one machine. It's amazing, really.
Now let's look at the way phpBB deals with searching. Every time a new post is made, it's split up into separate words and each word (except for really short ones and a bunch of very frequently used one, like "think", "fucking" and "cocks") is inserted into one big table. The purpose of this is to have an index of every single searchable word and then use it to speed up search queries.
For example, let's say the search word table looks like this:
id word
1 spore
2 katamari
3 damacy
4 metal
5 gear
6 solid
...
Then someone decides to make a new thread dedicated to Spore and posts "Hay guys, did you hear about this game Spore by Will Wright? It's going to be awesome." phpBB splits the post into words and first discards those that are 3 letters long or less. Then it discards the really popular ones that don't really help searching. So now we're left with "spore", "wright" and "awesome". phpBB then checks to see if every single one of those words is already in the big word table. Let's say "spore" is there, but "wright" and "awesome" are not. Now we need to add them:
id word
1 spore
2 katamari
3 damacy
4 metal
5 gear
6 solid
...
121446 wright
121447 awesome
Finally, we have another table to match words with certain posts. Let's say the above Spore post was #12345. So phpBB connects the three words with this post:
post word
...
12345 1
12345 121446
12345 121447
So when somebody makes a search for "will wright spore", phpBB quickly matches the words "wright" and "spore" (through their IDs) with post #12345 and outputs the results.
Now imagine how much load a single machine is under while trying to deal with multiple posts per second like that. Remember "could not insert word matches"? That's MySQL throwing in the towel. What phpBB does might have been a suitable solution for small-to-medium sized forums back before MySQL implemented full-text search, but it's clearly too inefficient for PA. Besides, like alpha said, MySQL simply lacks certain features that would have helped avoid table corruption.
The moral of the story is we better get ready to welcome our new Postgre overlords.
Whoa. Nice Post. That's 100% right. As a note of interest for devs out there, the full text searching in MySQL (atleast <= 4.0.x) does not scale well. Something external like zebra would work a lot better, but thats a big code change. BartVB and I have talked a bit about it over at phpBB.com, of their devs and support staff he is one of a handful that seems to know whats up.
I'm wondering why the the option of asking the membership for financial help in acquiring new and more servers was never put forth?
If I remember right, it's because we didn't want to place the onus of fixing the forums on the forumers.
I am pretty sure the problem would be someone acting up and bitching about getting banned even though they paid for the privilege of posting. Ya know, the kind of shit that TTB puts up with.
Basically. That and Gabe and Tycho don't feel like the forumers need to support the site financially like they used to back in the Club PA days. If I feel like post upgrade we need mor hardware, Robert and I will talk. Honest.
Posts
It seems some really major forums skate by on SQL.
it's not SQL itself, it's mySQL.
really big internet forums are usually not running off mySQL. there are exceptions. i've seen a huge forum once that was running an Oracle database, which I believe that if money was no object, would be the SQL database Alpha would like to run these forums off of.
Now let's look at the way phpBB deals with searching. Every time a new post is made, it's split up into separate words and each word (except for really short ones and a bunch of very frequently used one, like "think", "fucking" and "cocks") is inserted into one big table. The purpose of this is to have an index of every single searchable word and then use it to speed up search queries.
For example, let's say the search word table looks like this: Then someone decides to make a new thread dedicated to Spore and posts "Hay guys, did you hear about this game Spore by Will Wright? It's going to be awesome." phpBB splits the post into words and first discards those that are 3 letters long or less. Then it discards the really popular ones that don't really help searching. So now we're left with "spore", "wright" and "awesome". phpBB then checks to see if every single one of those words is already in the big word table. Let's say "spore" is there, but "wright" and "awesome" are not. Now we need to add them: Finally, we have another table to match words with certain posts. Let's say the above Spore post was #12345. So phpBB connects the three words with this post: So when somebody makes a search for "will wright spore", phpBB quickly matches the words "wright" and "spore" (through their IDs) with post #12345 and outputs the results.
Now imagine how much load a single machine is under while trying to deal with multiple posts per second like that. Remember "could not insert word matches"? That's MySQL throwing in the towel. What phpBB does might have been a suitable solution for small-to-medium sized forums back before MySQL implemented full-text search, but it's clearly too inefficient for PA. Besides, like alpha said, MySQL simply lacks certain features that would have helped avoid table corruption.
The moral of the story is we better get ready to welcome our new Postgre overlords.
so, my shitty inconsistent spelling make the forum run slower and kicks the search function in the nuts?
TMYK
jasconius(moderator);
I used to use the egosearch to keep track of stuff, though it doesn't track threads that we are interested in but haven't posted in yet.
Edit: Actually, would it be possible to keep the user searching active while disabling the word tracking until you get the new setup running?
There's 42k members on the member list, and while most inactive there's probably a solid number of lurkers as well who might help. Further I think if the financial situation was put in front of people, it would really make them think about how much we use this forum and might cut back on redundant or retarded threads. I mean if 10k people all gave about $10 in donations per year, that's a nice $100k. I'm willing to break into my piggy bank if it means getting standard features, and increased forum stability:
I think you're looking at more like $1000, if that.
I am pretty sure the problem would be someone acting up and bitching about getting banned even though they paid for the privilege of posting. Ya know, the kind of shit that TTB puts up with.
we need forum cleansing
Or maybe another postwhore hostage scenario
hopefully there will not be a repeat of last year
what did they do last year?
It definately wasnt good.
Because you were the one killing people.
Or am I thinking of the year before.
Actually, it was pretty fun barring that part of it
It would also be kind of cool to have a post count leak on the last day of this iteration of the forums.
edit: and no, I'm not speaking officially about the new software, I'm just applying some common sense + SQL knowledge.
Except for the bleading eyes.
Ramius' April Fools prank was top-notch.
the random avatars? That shit was wacky.
didn't we also look like TTB or something at one point?
conservation man, its the key
I think.
What the hell?