To be fair, according to Metacritic, it's wrong on several fronts: They don't have that many tiers, the differences between tiers is not as great, and some publications are not only wrong on the list, but flat missing.
The first two are the important ones to me. If Metacritic is internally using different tiers, and those tiers have a much smaller weight delta, then the data is completely, 100%, worthless, and if PR companies use it, they will fuck themselves.
On the other hand of course Metacritic will say it's wrong.....but where's the proof? They can show what they "use" but until the they show it we got no reason to believe the list is wrong.
Until I see the long form weight chart, I don't believe them.
You joke, but is the complete paper/talk with the full formula available anywhere?
0
Options
AegeriTiny wee bacteriumsPlateau of LengRegistered Userregular
I agree with The_Scarab: Even if the researchers aren't correct, if their method works and can produce a wildly different interpretation than the one Metacritic uses they should show how they rate things (in order to be as accurate as possible). Transparency is incredibly important here and if people suspect metacritic of wrong doing then that's a bad thing.
Zxerolfor the smaller pieces, my shovel wouldn't doso i took off my boot and used my shoeRegistered Userregular
Normally, all this hullabaloo wouldn't be worth the bits they were transmitted on, but when developers are now getting paid depending on their Metacritic scores and you have companies like Creative Assembly proudly declaring they have internal design metrics and goals to shape the best Metacritic score they can when they develop their titles, it becomes a much bigger deal.
Maybe that's less about Metacritic being more transparent on their methodology and more on publishers placing so much emphasis on it.
As I've mentioned, there's so many variables in how the buckets and weights could potentially work that there's a ton of possible solutions. Having one potential already shown false solution isn't enough to force their hand nor the fact that it isn't easily reverse engineerable. No one suspected metacritic of any wrongdoing until this list came out with shit that didn't make sense in it and a huge swing in weights that doesn't make sense to use. This whole thing reeks to me as an engineered crisis.
As we speak, statistics majors all around the country are working on confirming this data with a much larger sample size. Never underestimate how much fun it can be to be part of a gotcha moment.
If this was wrong, as in not even close to being true, Metacritic wouldn't have commented. Why would they? Dismiss it out of hand. I used to work in PR and I would always tell clients that you don't deny something a little, you deny everything unequivocally or you don't say anything at all. A half-denial only breeds suspicion and investigation.
They did deny it unequivocally. Did you read the post? It boils down to "he wrong, don't listen to him".
Again, people seem to think that Metacritic is under some moral obligation to give us the data, if for no other reason than to "prove the guy wrong". Zzzttt, no.
I am also curious what "gotcha" moment there is here. That Metacritic uses tiered weights? They've never denied it. Where is the gotcha? What lie is Matacritic being caught in?
The fact is Metacritic may as well have made an open challenge to hundreds of incredibly smart people, who will read the original paper (I assume that is somewhere?) and then confirm or deny what the other researchers found. While it is true the weights might be different, we'll have to see what the methodology of the paper in question is and how others react to it. Ultimately it doesn't matter jack shit what Metacritic says: It's going to be if other people can replicate the weightings that the original researchers found (which is how any science is done really). If Metacritic refuse to release their own numbers, they will have to live with what other people "confirm" to be their ratings numbers - because those doing this work will be the only ones providing evidence.
So my question is the talk anywhere? What was their method? How well did their method apply to X scores? Is any of this available?
Edit: Also, I may be contrary but I don't see anything wrong with a weighting system. How do you determine the value of a site that rates in 5 stars (as an example) vs. one with a 10 point scale? A 5 star review = 100 while getting a perfect 10 (even from IGN) is not anywhere near as easy. So you need some way to adjust these. I think their weighting is really bad if the researchers are correct, but a much smaller rating scale could account for that and even out the results.
The weighting isn't meant to convert a 5-point scale to a 100-point scale. It's meant to say "if these three sites give the game a 50, but IGN says it's a 100, then we'll call it even at 75".
If you want an intellectually honest interpretation of a valid use of a weighting system of this type, then think of it as the smaller sites representing the tastes of some smaller number of people. We've got SanFranciscoBlackPeople.com representing 60,000 people, and WhiteWyomingWomen.com representing a quarter million. If WhiteWyomingWomen.com gives Mormon Simulator 2014 a perfect score, while SanFranciscoBlackPeople.com gives it a 0, then we can say that the average person (assuming nobody else exists) might consider the game a solid 80.
However, I highly doubt the weighting system is used in this way, for various reasons including that it would be incredibly difficult to cut through all the layers of abstraction to actually get something useful out of it. You may as well just bullshit up some numbers and who's gonna know or care?
Is there any reason that PR firms/departments couldn't already produce these results? Given the amount of value placed on a metacritic score, what's to believe that PR somewhere couldn't/wouldn't contract their own statisticians to recreate Metacritic's weighting system (I'm going to go out on a limb and posit that it's absolutely reverse engineerable)? A margin for error will always exist, but any seemingly reliable results would surely influence exclusive previews, prerelease copy distribution, and ultimately a 'streamlining' of PR spending. This situation was an inevitability.
Regardless, the idea's been publicized and even Full Sail's model could produce a tangible change in gaming journalism, however marginal.
0
Options
HenroidMexican kicked from Immigration ThreadCentrism is Racism :3Registered Userregular
Their claim is that this whole thing is "inaccurate" rather than "false" or "untrue." Which means there is a weighting being applied to certain publications over others.
Their claim is that this whole thing is "inaccurate" rather than "false" or "untrue." Which means there is a weighting being applied to certain publications over others.
Is this in response to me?
0
Options
HenroidMexican kicked from Immigration ThreadCentrism is Racism :3Registered Userregular
Their claim is that this whole thing is "inaccurate" rather than "false" or "untrue." Which means there is a weighting being applied to certain publications over others.
Is this in response to me?
It's my response to Metacritic's lame defense.
0
Options
ShadowfireVermont, in the middle of nowhereRegistered Userregular
Their claim is that this whole thing is "inaccurate" rather than "false" or "untrue." Which means there is a weighting being applied to certain publications over others.
I was pretty sure they already discussed that years ago. Some proprietary formula or some such.
MordaRazgromМорда РазгромRuling the Taffer KingdomRegistered Userregular
My cynisism, looking at that list, has gone into over-drive. Granted I think reviews are absolutely worthless anyway, but valuing one over another...it just screams of favoritism and lack of independence. Granted, there are some that truly deserve to be weighted higher than others, but a lot of those sites...cynically, I just think "they didn't give them enough money" or "they didn't score games the way they were told to".
Monster Hunter Tri code/username: 1MF42Z (Morda)
WiiU Username: MordaRazgrom
Steam Username: MordaRazgrom
WoW/Diablo 3 Battlenet Battletag: MordaRazgrom#1755
Me and my wife have a gamer YouTube page if interested www.youtube.com/TeamMarriage
I don't get why this site isn't as simple as: take all the scores, add them up, and divide by the number of scores. Is that really so tough?
That method has problems though. For example, when you say 'all the scores', who is determining which scores count? Obviously IGN and Gamespot count. But does someone's tumblr blog or a wordpress startup? I mean, technically speaking, they have as much worth as anyone else's opinion. But by having no barrier to entry not only do you make it less valuable of an aggregate, but you also make it extremely vulnerable to sabotage, or a gaming of the system by publishers who seek to inflate their own scores.
So clearly Metacritic has to have some exclusivity to the list of scores that 'count'. But the various methods to do this also have problems. You can't make it based on site traffic, as that creates a snowball effect and a narrowing of opinion. You can't make it based on anything as crass as money, or a pay-to-count system. Because that results in the same thing; lots of power in a small number of places.
The truth is that there is no easy way to do it. But the way that it is done should be mathematical. It should include a multitude of methods, all contributing towards a complete weighting. Site traffic alone would poison the industry, as I said. But I do think that site traffic should count, at least a little. This is, after all, a score aggregate, whose purpose is to represent popular opinion quickly and easily. One could argue that site traffic is an indicator of customer approval. People often read reviews that agree with their own opinion, while shunning those that don't. There are caveats to this oversimplification, of course, which is why Metacritic should take into account other things. Such as the timing of reviews, the normal distribution of scores, the comparison of review scores to user scores, and so on.
The point being that however complex and convoluted this method seems, it is all purely mathematical and therefore, with a bit of work, can be reverse engineered. Or, at the very least, an approximation can be made. But the most damning thing of this whole story is that in their own denial of the proposed weightings, Metacritic admitted to an even worse method of calculating their scores.
Look at the last line of their rebuttal:
In addition, our weights are periodically adjusted as needed if, over time, a publication demonstrates an increase or decrease in overall quality.
Overall quality is not a measurable quantity. It is a subjective, and rather vague, point of opinion. What Metacritic has just said is that yes, we are ultimately no more scientific than the guy on the street. The quality of a gaming site should never factor into a review aggregation. That is, in of itself, an act of subjective control. The whole point of Metacritic is to eliminate as much bias and opinion as possible, in order to provide a more general, objective response. Naturally you can't have an objective score for a work of art, but you can get a general consensus through mathematical averaging, as you said. That Metacritic keeps their system a closely guarded secret is admitting that somewhere down the line someone is making a decision. There is someone in their offices choosing what counts more than the rest. Otherwise, they would have no need to keep their method secret. I don't need to tell you how sinister this shit is.
I firmly believe that everyone's opinion should be treated equally for the purposes of review aggregation. Otherwise what is the point of aggregating the reviews? Why not just say 'IGN's review of Tomb Raider is the one we agree with most, read that'. Startup sites with no money and no readership can, and often do, offer just as much insight and informed opinion on games as the biggest of multinational juggernauts. And I believe that some moron's misguided and poorly thought out dross should not be treated differently than any other. Certainly not by Metacritic. Let the readers decide for themselves with click-throughs and comment threads, otherwise why do you exist in the first place?
The fact that Metacritic is owned by the same company that owns Gamespot should say it all, really. They keep their methodology a secret because they are just another marketing tool of the industry. Even if those proposed weightings are completely off, the fact that the whole internet was not surprised by them shows how little credibility that site has. Divorcing Metacritic scores from bonus pay and work contracts should be at the top of everyone's agenda's right now. It is a poisonous, insidious system of control that is doing real harm to real, hard-working people. And the first step to achieving that goal should be making these obscenely important Metacritic scores a transparent, easily understood process so that at least it is fair.
Posts
You joke, but is the complete paper/talk with the full formula available anywhere?
Maybe that's less about Metacritic being more transparent on their methodology and more on publishers placing so much emphasis on it.
The weighting isn't meant to convert a 5-point scale to a 100-point scale. It's meant to say "if these three sites give the game a 50, but IGN says it's a 100, then we'll call it even at 75".
If you want an intellectually honest interpretation of a valid use of a weighting system of this type, then think of it as the smaller sites representing the tastes of some smaller number of people. We've got SanFranciscoBlackPeople.com representing 60,000 people, and WhiteWyomingWomen.com representing a quarter million. If WhiteWyomingWomen.com gives Mormon Simulator 2014 a perfect score, while SanFranciscoBlackPeople.com gives it a 0, then we can say that the average person (assuming nobody else exists) might consider the game a solid 80.
However, I highly doubt the weighting system is used in this way, for various reasons including that it would be incredibly difficult to cut through all the layers of abstraction to actually get something useful out of it. You may as well just bullshit up some numbers and who's gonna know or care?
Regardless, the idea's been publicized and even Full Sail's model could produce a tangible change in gaming journalism, however marginal.
Their claim is that this whole thing is "inaccurate" rather than "false" or "untrue." Which means there is a weighting being applied to certain publications over others.
Is this in response to me?
It's my response to Metacritic's lame defense.
I was pretty sure they already discussed that years ago. Some proprietary formula or some such.
Hint: probably rhymes with 'collars'.
Currently playing: GW2 and TSW
Metacritic is supposed to offer you a single score that more accurately reflects the trend of all critics' opinion and its influence.
Getting a 10 on "I love pencils" blog that was started last week cannot be as relevant as getting a 5 on Eurogamer.
WiiU Username: MordaRazgrom
Steam Username: MordaRazgrom
WoW/Diablo 3 Battlenet Battletag: MordaRazgrom#1755
Me and my wife have a gamer YouTube page if interested www.youtube.com/TeamMarriage
That method has problems though. For example, when you say 'all the scores', who is determining which scores count? Obviously IGN and Gamespot count. But does someone's tumblr blog or a wordpress startup? I mean, technically speaking, they have as much worth as anyone else's opinion. But by having no barrier to entry not only do you make it less valuable of an aggregate, but you also make it extremely vulnerable to sabotage, or a gaming of the system by publishers who seek to inflate their own scores.
So clearly Metacritic has to have some exclusivity to the list of scores that 'count'. But the various methods to do this also have problems. You can't make it based on site traffic, as that creates a snowball effect and a narrowing of opinion. You can't make it based on anything as crass as money, or a pay-to-count system. Because that results in the same thing; lots of power in a small number of places.
The truth is that there is no easy way to do it. But the way that it is done should be mathematical. It should include a multitude of methods, all contributing towards a complete weighting. Site traffic alone would poison the industry, as I said. But I do think that site traffic should count, at least a little. This is, after all, a score aggregate, whose purpose is to represent popular opinion quickly and easily. One could argue that site traffic is an indicator of customer approval. People often read reviews that agree with their own opinion, while shunning those that don't. There are caveats to this oversimplification, of course, which is why Metacritic should take into account other things. Such as the timing of reviews, the normal distribution of scores, the comparison of review scores to user scores, and so on.
The point being that however complex and convoluted this method seems, it is all purely mathematical and therefore, with a bit of work, can be reverse engineered. Or, at the very least, an approximation can be made. But the most damning thing of this whole story is that in their own denial of the proposed weightings, Metacritic admitted to an even worse method of calculating their scores.
Look at the last line of their rebuttal:
Overall quality is not a measurable quantity. It is a subjective, and rather vague, point of opinion. What Metacritic has just said is that yes, we are ultimately no more scientific than the guy on the street. The quality of a gaming site should never factor into a review aggregation. That is, in of itself, an act of subjective control. The whole point of Metacritic is to eliminate as much bias and opinion as possible, in order to provide a more general, objective response. Naturally you can't have an objective score for a work of art, but you can get a general consensus through mathematical averaging, as you said. That Metacritic keeps their system a closely guarded secret is admitting that somewhere down the line someone is making a decision. There is someone in their offices choosing what counts more than the rest. Otherwise, they would have no need to keep their method secret. I don't need to tell you how sinister this shit is.
I firmly believe that everyone's opinion should be treated equally for the purposes of review aggregation. Otherwise what is the point of aggregating the reviews? Why not just say 'IGN's review of Tomb Raider is the one we agree with most, read that'. Startup sites with no money and no readership can, and often do, offer just as much insight and informed opinion on games as the biggest of multinational juggernauts. And I believe that some moron's misguided and poorly thought out dross should not be treated differently than any other. Certainly not by Metacritic. Let the readers decide for themselves with click-throughs and comment threads, otherwise why do you exist in the first place?
The fact that Metacritic is owned by the same company that owns Gamespot should say it all, really. They keep their methodology a secret because they are just another marketing tool of the industry. Even if those proposed weightings are completely off, the fact that the whole internet was not surprised by them shows how little credibility that site has. Divorcing Metacritic scores from bonus pay and work contracts should be at the top of everyone's agenda's right now. It is a poisonous, insidious system of control that is doing real harm to real, hard-working people. And the first step to achieving that goal should be making these obscenely important Metacritic scores a transparent, easily understood process so that at least it is fair.