The new forums will be named Coin Return (based on the most recent vote)! You can check on the status and timeline of the transition to the new forums here.
The Guiding Principles and New Rules document is now in effect.
Metacritic Weighting Scale revealed
Allforce
Registered User regular
Registered User regular
Cade posted in the industry thread but this is really bigger news than that thread can stomach.
The website Metacritic weights the scores of different video game critics and publications when calculating its aggregate 'metascore' for any particular title.
Each critic/ publication is assigned one of six different weightings of 'importance', with some publications exerting considerably more influence over a game's final 'metascore' than others.
This was revealed by Adams Greenwood-Ericksen of Full Sail University at a talk titled 'A Scientific Assessment of the Validity and Value of Metacritic' delivered at the Game Developer's Conference in San Fransisco this afternoon.
Metacritic confirmed to Greenwood-Ericksen during the course of his research that the site applies different weightings to incoming critics and publications' reviews in order to calculate its 'averaged' numerical score for any particular title.
However, it would not reveal how the weightings were assigned to different publications or on what criteria one critic was given a greater weighing than another.
The course director and his students then set about modelling the weightings based on data pulled from the site. Finally, after six months of work, the researchers compared their modeled scores to the actual scores and discovered that across the 188 publications that feed into Metacritic's video game score work, their findings were almost entirely accurate.
Greenwood-Ericksen stated they wanted to carry out the research as Metacritic scores are "very important to a lot of people" and pointed out that, when publishers withhold financial bonuses when a game doesn't reach its Metacritic target, livelihoods are tied up in the site's work.
He also reminded attendees that often a publisher's Wall Street stock can change on the basis of a Metacritic score., and as such the site's workings are of practical interest.
The findings will also be of interest to consumers as, if accurate, they reveal that some official magazines and sites (which are sponsored by platform holders in some cases) are assigned a greater weighting than independent sites and critics.
Metacritics weighting system revealed.
For length:Here is the full listing of score weightings used by Metacritic according to Greenwood-Ericksen's research:
Weighting -- Critic/ Publication
Highest (1.5) -- Dark Zero
Highest (1.5) -- Digital Chumps
Highest (1.5) -- Digital Entertainment News
Highest (1.5) -- Extreme Gamer
Highest (1.5) -- Firing Squad
Highest (1.5) -- Game Almighty
Highest (1.5) -- Game Informer
Highest (1.5) -- GamePro
Highest (1.5) -- Gamers Europe
Highest (1.5) -- GameTrailers
Highest (1.5) -- GotNext
Highest (1.5) -- IGN
Highest (1.5) -- IGN AU
Highest (1.5) -- IGN UK
Highest (1.5) -- Just Adventure
Highest (1.5) -- Machinima
Highest (1.5) -- Planet Xbox 360
Highest (1.5) -- PlayStation Official Magazine UK
Highest (1.5) -- PlayStation Official Magazine US
Highest (1.5) -- Telegraph
Highest (1.5) -- The New York Times
Highest (1.5) -- TheSixthAxis
Highest (1.5) -- TotalPlayStation
Highest (1.5) -- VGPub
Highest (1.5) -- Videogameszone.de
Highest (1.5) -- Wired
Highest (1.5) -- Xboxic
Highest (1.5) -- Yahoo Games
Highest (1.5) -- ZTGames Domain
High (1.25) -- Absolute Games
High (1.25) -- ActionTrip
High (1.25) -- Adventure Gamers
High (1.25) -- Computer & Video Games
High (1.25) -- Console Gameworld
High (1.25) -- Da GameBoyz
High (1.25) -- Darkstation
High (1.25) -- Edge Magazine
High (1.25) -- EGM
High (1.25) -- EuroGamer Italy
High (1.25) -- EuroGamer Spain
High (1.25) -- G4 TV
High (1.25) -- Game Chronicles
High (1.25) -- GameDaily
High (1.25) -- Gameplayer
High (1.25) -- Gamer 2.0
High (1.25) -- Gamervision
High (1.25) -- Games Master UK
High (1.25) -- Gamespot
High (1.25) -- GameSpy
High (1.25) -- Gaming Age
High (1.25) -- Gaming Nexus
High (1.25) -- Maxi Consoles (Portugal)
High (1.25) -- Pelit
High (1.25) -- Play.tm
High (1.25) -- PlayStation Universe
High (1.25) -- PlayStation Official AU
High (1.25) -- PSM3 Magazine UK
High (1.25) -- PS Extreme
High (1.25) -- RPG Fan
High (1.25) -- Strategy Informer
High (1.25) -- Team Xbox
High (1.25) -- The Onion (AV Club)
High (1.25) -- Totally 360
High (1.25) -- WonderwallWeb
High (1.25) -- XGN
Medium (1.0) -- 1Up
Medium (1.0) -- CPU Gamer
Medium (1.0) -- Cubed3
Medium (1.0) -- Cynamite
Medium (1.0) -- D+Pad Magazine
Medium (1.0) -- DailyGame
Medium (1.0) -- Destructoid
Medium (1.0) -- Eurogamer
Medium (1.0) -- Everyeye.it
Medium (1.0) -- Game Revolution
Medium (1.0) -- Game Shark
Medium (1.0) -- Gameblog.fr
Medium (1.0) -- GameKult
Medium (1.0) -- Gamereactor Denmark
Medium (1.0) -- Gamers' Temple
Medium (1.0) -- GameShark
Medium (1.0) -- Gameblog.fr
Medium (1.0) -- GamesNation
Medium (1.0) -- GameStar
Medium (1.0) -- GameTap
Medium (1.0) -- Gaming Target
Medium (1.0) -- Gamereactor Sweden
Medium (1.0) -- The Guardian
Medium (1.0) -- Hardcore Gamer Magazine
Medium (1.0) -- HellBored
Medium (1.0) -- NiceGamers
Medium (1.0) -- Joystiq
Medium (1.0) -- Just RPG
Medium (1.0) -- Level 7.nu
Medium (1.0) -- Modojo
Medium (1.0) -- MondoXbox
Medium (1.0) -- Multiplayer.it
Medium (1.0) -- N-Europe
Medium (1.0) -- Netjak
Medium (1.0) -- NGamer Magazine
Medium (1.0) -- Nintendo Life
Medium (1.0) -- Nintendo Power
Medium (1.0) -- Nintendojo
Medium (1.0) -- Nintendo World Report
Medium (1.0) -- NZGamer
Medium (1.0) -- Official Nintendo Magazine UK
Medium (1.0) -- Official Xbox 360 Magazine UK
Medium (1.0) -- Official Xbox Magazine
Medium (1.0) -- Official Xbox Magazine UK
Medium (1.0) -- PALGN
Medium (1.0) -- PC Format
Medium (1.0) -- PC Gamer (Germany)
Medium (1.0) -- PC Gamer UK
Medium (1.0) -- PC Gamer
Medium (1.0) -- PC Powerplay
Medium (1.0) -- PGNx Media
Medium (1.0) -- Play Magazine
Medium (1.0) -- PlayStation LifeStyle
Medium (1.0) -- Pocketgamer UK
Medium (1.0) -- PT Games
Medium (1.0) -- Real Gamer
Medium (1.0) -- SpazioGames
Medium (1.0) -- Talk Xbox
Medium (1.0) -- The Escapist
Medium (1.0) -- Thunderbolt
Medium (1.0) -- Total VideoGames
Medium (1.0) -- Worth Playing
Medium (1.0) -- X360 Magazine UK
Medium (1.0) -- Xbox World 360 Magazine UK
Medium (1.0) -- Xbox World Australia
Medium (1.0) -- Xbox360 Achievements
Medium (1.0) -- Xbox Addict
Low (0.75) -- 360 Gamer Magazine UK
Low (0.75) -- 3DJuegos
Low (0.75) -- Ace Gamez
Low (0.75) -- Atomic Gamer
Low (0.75) -- BigPond GameArena
Low (0.75) -- Console Monster
Low (0.75) -- Deeko
Low (0.75) -- Eurogamer Portugal
Low (0.75) -- Game Focus
Low (0.75) -- Gameplanet
Low (0.75) -- Gamer Limit
Low (0.75) -- Gamer.nl
Low (0.75) -- Games Radar (in-house)
Low (0.75) -- Games TM
Low (0.75) -- Gamestyle
Low (0.75) -- GameZone
Low (0.75) -- Gaming Excellence
Low (0.75) -- Gaming Trend
Low (0.75) -- Impulse gamer
Low (0.75) -- Kombo
Low (0.75) -- MEGamers
Low (0.75) -- Metro Game Central
Low (0.75) -- MS Xbox World
Low (0.75) -- NTSC-uk
Low (0.75) -- PS Focus
Low (0.75) -- PSW Magazine UK
Low (0.75) -- Video Game Talk
Low (0.75) -- VideoGamer
Lower (0.5) -- Armchair Empire
Lower (0.5) -- Cheat Code Central
Lower (0.5) -- Game Over Online
Lower (0.5) -- Game Positive
Lower (0.5) -- Gamer's Hell
Lower (0.5) -- Gamereactor Sweden
Lower (0.5) -- Gamers.at
Lower (0.5) -- Giant Bomb
Lower (0.5) -- PS3bloggen.se
Lower (0.5) -- RPGamer
Lower (0.5) -- Vandal Online
Lowest (0.25) -- 9Lives
Lowest (0.25) -- Boomtown
Lowest (0.25) -- Computer Games Online RO
Lowest (0.25) -- GamerNode
Lowest (0.25) -- GamingXP
Lowest (0.25) -- IC-Games
Lowest (0.25) -- Insidegamer.nl
Lowest (0.25) -- Jolt Online Gaming
Lowest (0.25) -- Kikizo
Lowest (0.25) -- LEVEL
Lowest (0.25) -- Meritstation
Lowest (0.25) -- My Gamer
Lowest (0.25) -- Official PlayStation 2 Magazine UK
Lowest (0.25) -- Play UK
Lowest (0.25) -- WHAM! Gaming
0
Posts
Then again, my memory is pretty bad.
[Edit]
Also at the PAX panel Gerstmann basically said that they mostly do reviews so that publishers will send them pre-release copies of games because they are on Metacritic.
(Please do not gift. My game bank is already full.)
This can't possibly end well.
I guess they're also weighted low because of their scale being five point rather than the normal hundred/twenty/ten scale. Less information, less distinction between titles.
At the very least, this spells a big shakeup because which publisher is going to spend money courting a 0.5 site instead of a 1.5?
The whole list is.....nuts. I can't see a lot of the sites that do reviews being very happy and those that are listed at the top are bound to be highly sought after by publishers while others feel like they've been jerked around until now, unless they knew which is definitely possible.
And I wonder how often are these rankings changed. I have a feeling that FiringSquad isn't as big as it used to be.
(Please do not gift. My game bank is already full.)
(Please do not gift. My game bank is already full.)
I hadn't even heard of Dark Zero before this list was published. It also makes me wonder if there is anything nefarious at all here. If money was changing hands to get these ratings, I think the list would look much different.
This again rolls me back to my original point: Without knowing how these ratings are arrived at, it's just noise. It gives us an idea of how Metacritic scores are calculated, but not much else. We also have no idea how fluid the list is.
https://www.facebook.com/Metacritic/posts/501424766586647
To be fair, more people in the world will get their game reviews from NYT than from any hardcore gaming site, like Giant Bomb or Gamespot.
Metacritic isn't just about games. It has a very broad demographic.
I'm just sayin'.
Yeah I'm with you on this.
The idea of weighted scores doesn't exactly shock me or outrage me. If anything it makes sense (depending on how the weighting is done). I mean, IMHO, it'd make sense that the review/score from a "respected" source would carry more weight then the review of some random dude from some random site.
However, obviously that isn't how they have determined the "weight" of each site. Since I too have never, ever, heard of Dark Zero and if Domstercool is to be believed (which I see no reason to doubt him) it is a site comprised of random Joes who like video games.
Although to be fair, those are the kind of opinions I prefer. :P
edit- And I don't mean to single out Dark Zero. There are a fair number of sites on that list that I've never heard of who rank higher then other sites I'd describe as "household names" among the gaming community.
Yeah I get that exposure is a big factor here, with something like Yahoo games in the top tier too, still funny though.
What's weirder though is the sites in the top tier I have never heard of. Dark Zero, Got Next? Maybe I stick to my internet comfort zone a bit much but no idea who they are.
Yeah, I'm not trying to single out Dark Zero, it was just the first on the high list, and jumped out at me as "Who?".
Exactly this.
Issue 1: The list is in the open. The weighting of scores is essentially going to become a weighting of marketing dollars put towards each publication. This is a roadmap for PR.
Issue 2: The seemingly nonsensical weighting differences between strange publications.
The first issue is much, much more meaningful and important. And more damaging to Metacritic. The second may well turn out to be a non-issue entirely. We don't yet know the way it is calculated (and it could be completely reasonable).
The first two are the important ones to me. If Metacritic is internally using different tiers, and those tiers have a much smaller weight delta, then the data is completely, 100%, worthless, and if PR companies use it, they will fuck themselves.
On the other hand of course Metacritic will say it's wrong.....but where's the proof? They can show what they "use" but until the they show it we got no reason to believe the list is wrong.
Indeed.
I'm not exactly a fan of Metacritic myself, for various reason we've gone over at length on this forum, so if anything I guess I hope that this destroys Metacritic's credibility and people's jobs/bonuses are no longer on the line because a reviewer was ordered to review a game from a genre he wasn't a fan of.
Though I suppose that is wishful thinking.
@Cade
It is catch-22. If they do publish the list to show that the other one is wrong, then we are back to Issue 1. If they don't then Issue 1 is still a problem, only more harmful to all involved I guess.
You also have no reason to believe it's right. Grain of salt and all that.
It's also not Metacritics responsibility to prove some researcher wrong, not in the slightest. They aren't under some order to give us the data, and they likely never will (for the exact reasons that Scarab pointed out, because it would open a huge can of worms).
I'm guessing no matter what Metacritic says to the contrary their actual list of tiered sites aren't that far off from what this list actually is, since it's all computed from what I can only imagine is roughly the same pool of data.
If this was wrong, as in not even close to being true, Metacritic wouldn't have commented. Why would they? Dismiss it out of hand. I used to work in PR and I would always tell clients that you don't deny something a little, you deny everything unequivocally or you don't say anything at all. A half-denial only breeds suspicion and investigation.
They did deny it unequivocally. Did you read the post? It boils down to "he wrong, don't listen to him".
Again, people seem to think that Metacritic is under some moral obligation to give us the data, if for no other reason than to "prove the guy wrong". Zzzttt, no.
I am also curious what "gotcha" moment there is here. That Metacritic uses tiered weights? They've never denied it. Where is the gotcha? What lie is Matacritic being caught in?
The reason they aren't listed is because the study to aggregate the weighing system started before Polygon was a thing / had enough reviews to qualify / whatever technicality. Polygon is a Metacritic approved whatever and whatnot.
The granularity of their data. Their slogan is 'Keeping score of entertainment'. If that score is reached on the back of a napkin, it severely damages their reputation. The seeming nonsensical nature of the weighting implies no cohesive method.
Furthermore, their statement uses clever vagaries to sidestep the issue. Of course they're not going to release their actual method as negative proof, but they offer no conclusive statement on the overall ethics of a hidden weighting system. Which was the point of the research talk in the first place. People's jobs are on the line and Metacritic is needlessly secretive on that fact.
Speculation is moot, however. Maths will win the day. Eventually. If a lie forces transparency for a poisonous industry juggernaut, then so be it.
Well the gotcha is that the weights make absolutely no sense.
But that is also why I kind of believe them when they say that this list is all wrong.
(Please do not gift. My game bank is already full.)
This is basically a nonstory unless you didn't know metacritic weighted review scores, then at least it informed you about that.
That is, in fact, how the entire branch of all human mathematics and science was created.
This isn't some infinite pandoras box with an impossible imaginary number. This is like, math, with a couple hundred thousand sources and a reasonably small number of pool entries.
There isn't infinite functions because there aren't infinite entries. Your entries are finite both in the numerical data and the pool the data is being pulled from (Review scores and reviewers.)
The only way you could literally not math this out is if they just rolled a die and reweighted every publication for every new entry for shits and giggles. Which is entirely possible as well.
Until I see the long form weight chart, I don't believe them.
https://steamcommunity.com/profiles/76561197970666737/
I may not agree with some of their reviews, but I have tremendous respect for Polygon. They run some great long-form articles, and are pretty well written. And the change in SimCity's review score is fine by me... Call a turd a turd if it's true, and if the reality of a game after launch doesn't line up with what they saw beforehand, a change is warranted.
Edit: I guess I just don't get the Polygon hate.
https://steamcommunity.com/profiles/76561197970666737/
It's a bummer to have to operate with a degree of cynicism at all times but let's be honest; every one of the cultural industries (film, TV, music, etc.) is at least this exploitative.
I guess I also don't see a huge deal here. There had to be a weight system in place for aggregating review scores, and they've admitted as much before. We don't know why different sites receive the weights they do, but it could be nothing shady at all, more based on their review score setup and how it meshes with metacritic's aggregate scores. Otherwise, Giant Bomb would throw review averages off considerably with their 5 point scale.
I mean, metacritic is still terrible, and so that is definitely a "this fucking industry" thing, but this reveal, if it is one (it probably is) probably isn't a huge deal.
https://steamcommunity.com/profiles/76561197970666737/
The structure and weighting of the buckets are a form of infinity in this case. Maybe the buckets aren't universal but are different by genre or something else? Maybe the buckets are universal but the weights are applied differently based on genre? Maybe there are no buckets and each site is given an individual weight? What if the weights change over time as a site becomes more popular or puts out more reviews? Are the scores locked in for a publication such that if their weight changes their old scores are still weighed by the previous weight or does their new weight retroactively apply?
Basically there's too many possibilities left open in the data we have to reliably reverse engineer their weighting system. If you assume Metacritic isn't lying when they say the bucket/weight structure these guys used was incorrect, then the list is invalid.
The thing about having different tiers or less tiers or less weighting really don't mean a ton in the end because the sample they tested from came to these conclusions.. which means their model is similar enough to the real thing that it can be used to game the system. I'm also willing to bet the publishers have access to the real numbers, or at least the big guys do.
If Metacritic's weighting is so subjective, why do they exist at all? The whole point of a meta-critic is to attempt to reduce subjectivity by applying the rules of crowd opinion. Now it turns out their weightings are periodically 'adjusted' depending on a publication's overall quality. So now the meta-critics are critiquing the critics themselves. We're through the fucking looking glass here, goddam.