As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/

Metacritic Weighting Scale revealed

AllforceAllforce Registered User regular
Cade posted in the industry thread but this is really bigger news than that thread can stomach.
The website Metacritic weights the scores of different video game critics and publications when calculating its aggregate 'metascore' for any particular title.

Each critic/ publication is assigned one of six different weightings of 'importance', with some publications exerting considerably more influence over a game's final 'metascore' than others.

This was revealed by Adams Greenwood-Ericksen of Full Sail University at a talk titled 'A Scientific Assessment of the Validity and Value of Metacritic' delivered at the Game Developer's Conference in San Fransisco this afternoon.

Metacritic confirmed to Greenwood-Ericksen during the course of his research that the site applies different weightings to incoming critics and publications' reviews in order to calculate its 'averaged' numerical score for any particular title.

However, it would not reveal how the weightings were assigned to different publications or on what criteria one critic was given a greater weighing than another.

The course director and his students then set about modelling the weightings based on data pulled from the site. Finally, after six months of work, the researchers compared their modeled scores to the actual scores and discovered that across the 188 publications that feed into Metacritic's video game score work, their findings were almost entirely accurate.

Greenwood-Ericksen stated they wanted to carry out the research as Metacritic scores are "very important to a lot of people" and pointed out that, when publishers withhold financial bonuses when a game doesn't reach its Metacritic target, livelihoods are tied up in the site's work.

He also reminded attendees that often a publisher's Wall Street stock can change on the basis of a Metacritic score., and as such the site's workings are of practical interest.

The findings will also be of interest to consumers as, if accurate, they reveal that some official magazines and sites (which are sponsored by platform holders in some cases) are assigned a greater weighting than independent sites and critics.
Cade wrote: »
Metacritics weighting system revealed.

For length:
Here is the full listing of score weightings used by Metacritic according to Greenwood-Ericksen's research:

Weighting -- Critic/ Publication

Highest (1.5) -- Dark Zero
Highest (1.5) -- Digital Chumps
Highest (1.5) -- Digital Entertainment News
Highest (1.5) -- Extreme Gamer
Highest (1.5) -- Firing Squad
Highest (1.5) -- Game Almighty
Highest (1.5) -- Game Informer
Highest (1.5) -- GamePro
Highest (1.5) -- Gamers Europe
Highest (1.5) -- GameTrailers
Highest (1.5) -- GotNext
Highest (1.5) -- IGN
Highest (1.5) -- IGN AU
Highest (1.5) -- IGN UK
Highest (1.5) -- Just Adventure
Highest (1.5) -- Machinima
Highest (1.5) -- Planet Xbox 360
Highest (1.5) -- PlayStation Official Magazine UK
Highest (1.5) -- PlayStation Official Magazine US
Highest (1.5) -- Telegraph
Highest (1.5) -- The New York Times
Highest (1.5) -- TheSixthAxis
Highest (1.5) -- TotalPlayStation
Highest (1.5) -- VGPub
Highest (1.5) -- Videogameszone.de
Highest (1.5) -- Wired
Highest (1.5) -- Xboxic
Highest (1.5) -- Yahoo Games
Highest (1.5) -- ZTGames Domain

High (1.25) -- Absolute Games
High (1.25) -- ActionTrip
High (1.25) -- Adventure Gamers
High (1.25) -- Computer & Video Games
High (1.25) -- Console Gameworld
High (1.25) -- Da GameBoyz
High (1.25) -- Darkstation
High (1.25) -- Edge Magazine
High (1.25) -- EGM
High (1.25) -- EuroGamer Italy
High (1.25) -- EuroGamer Spain
High (1.25) -- G4 TV
High (1.25) -- Game Chronicles
High (1.25) -- GameDaily
High (1.25) -- Gameplayer
High (1.25) -- Gamer 2.0
High (1.25) -- Gamervision
High (1.25) -- Games Master UK
High (1.25) -- Gamespot
High (1.25) -- GameSpy
High (1.25) -- Gaming Age
High (1.25) -- Gaming Nexus
High (1.25) -- Maxi Consoles (Portugal)
High (1.25) -- Pelit
High (1.25) -- Play.tm
High (1.25) -- PlayStation Universe
High (1.25) -- PlayStation Official AU
High (1.25) -- PSM3 Magazine UK
High (1.25) -- PS Extreme
High (1.25) -- RPG Fan
High (1.25) -- Strategy Informer
High (1.25) -- Team Xbox
High (1.25) -- The Onion (AV Club)
High (1.25) -- Totally 360
High (1.25) -- WonderwallWeb
High (1.25) -- XGN

Medium (1.0) -- 1Up
Medium (1.0) -- CPU Gamer
Medium (1.0) -- Cubed3
Medium (1.0) -- Cynamite
Medium (1.0) -- D+Pad Magazine
Medium (1.0) -- DailyGame
Medium (1.0) -- Destructoid
Medium (1.0) -- Eurogamer
Medium (1.0) -- Everyeye.it
Medium (1.0) -- Game Revolution
Medium (1.0) -- Game Shark
Medium (1.0) -- Gameblog.fr
Medium (1.0) -- GameKult
Medium (1.0) -- Gamereactor Denmark
Medium (1.0) -- Gamers' Temple
Medium (1.0) -- GameShark
Medium (1.0) -- Gameblog.fr
Medium (1.0) -- GamesNation
Medium (1.0) -- GameStar
Medium (1.0) -- GameTap
Medium (1.0) -- Gaming Target
Medium (1.0) -- Gamereactor Sweden
Medium (1.0) -- The Guardian
Medium (1.0) -- Hardcore Gamer Magazine
Medium (1.0) -- HellBored
Medium (1.0) -- NiceGamers
Medium (1.0) -- Joystiq
Medium (1.0) -- Just RPG
Medium (1.0) -- Level 7.nu
Medium (1.0) -- Modojo
Medium (1.0) -- MondoXbox
Medium (1.0) -- Multiplayer.it
Medium (1.0) -- N-Europe
Medium (1.0) -- Netjak
Medium (1.0) -- NGamer Magazine
Medium (1.0) -- Nintendo Life
Medium (1.0) -- Nintendo Power
Medium (1.0) -- Nintendojo
Medium (1.0) -- Nintendo World Report
Medium (1.0) -- NZGamer
Medium (1.0) -- Official Nintendo Magazine UK
Medium (1.0) -- Official Xbox 360 Magazine UK
Medium (1.0) -- Official Xbox Magazine
Medium (1.0) -- Official Xbox Magazine UK
Medium (1.0) -- PALGN
Medium (1.0) -- PC Format
Medium (1.0) -- PC Gamer (Germany)
Medium (1.0) -- PC Gamer UK
Medium (1.0) -- PC Gamer
Medium (1.0) -- PC Powerplay
Medium (1.0) -- PGNx Media
Medium (1.0) -- Play Magazine
Medium (1.0) -- PlayStation LifeStyle
Medium (1.0) -- Pocketgamer UK
Medium (1.0) -- PT Games
Medium (1.0) -- Real Gamer
Medium (1.0) -- SpazioGames
Medium (1.0) -- Talk Xbox
Medium (1.0) -- The Escapist
Medium (1.0) -- Thunderbolt
Medium (1.0) -- Total VideoGames
Medium (1.0) -- Worth Playing
Medium (1.0) -- X360 Magazine UK
Medium (1.0) -- Xbox World 360 Magazine UK
Medium (1.0) -- Xbox World Australia
Medium (1.0) -- Xbox360 Achievements
Medium (1.0) -- Xbox Addict

Low (0.75) -- 360 Gamer Magazine UK
Low (0.75) -- 3DJuegos
Low (0.75) -- Ace Gamez
Low (0.75) -- Atomic Gamer
Low (0.75) -- BigPond GameArena
Low (0.75) -- Console Monster
Low (0.75) -- Deeko
Low (0.75) -- Eurogamer Portugal
Low (0.75) -- Game Focus
Low (0.75) -- Gameplanet
Low (0.75) -- Gamer Limit
Low (0.75) -- Gamer.nl
Low (0.75) -- Games Radar (in-house)
Low (0.75) -- Games TM
Low (0.75) -- Gamestyle
Low (0.75) -- GameZone
Low (0.75) -- Gaming Excellence
Low (0.75) -- Gaming Trend
Low (0.75) -- Impulse gamer
Low (0.75) -- Kombo
Low (0.75) -- MEGamers
Low (0.75) -- Metro Game Central
Low (0.75) -- MS Xbox World
Low (0.75) -- NTSC-uk
Low (0.75) -- PS Focus
Low (0.75) -- PSW Magazine UK
Low (0.75) -- Video Game Talk
Low (0.75) -- VideoGamer

Lower (0.5) -- Armchair Empire
Lower (0.5) -- Cheat Code Central
Lower (0.5) -- Game Over Online
Lower (0.5) -- Game Positive
Lower (0.5) -- Gamer's Hell
Lower (0.5) -- Gamereactor Sweden
Lower (0.5) -- Gamers.at
Lower (0.5) -- Giant Bomb
Lower (0.5) -- PS3bloggen.se
Lower (0.5) -- RPGamer
Lower (0.5) -- Vandal Online

Lowest (0.25) -- 9Lives
Lowest (0.25) -- Boomtown
Lowest (0.25) -- Computer Games Online RO
Lowest (0.25) -- GamerNode
Lowest (0.25) -- GamingXP
Lowest (0.25) -- IC-Games
Lowest (0.25) -- Insidegamer.nl
Lowest (0.25) -- Jolt Online Gaming
Lowest (0.25) -- Kikizo
Lowest (0.25) -- LEVEL
Lowest (0.25) -- Meritstation
Lowest (0.25) -- My Gamer
Lowest (0.25) -- Official PlayStation 2 Magazine UK
Lowest (0.25) -- Play UK
Lowest (0.25) -- WHAM! Gaming

«1

Posts

  • lowlylowlycooklowlylowlycook Registered User regular
    edited March 2013
    Hmmmm, I kind or remember the guys from IGN complaining that publishers they talked to would ask, "Are you from IGN or Gamespot?" and become pretty uninterested when the answer was no. It seems odd to me that Gamespot wouldn't be in the highest category.

    Then again, my memory is pretty bad.

    [Edit]

    Also at the PAX panel Gerstmann basically said that they mostly do reviews so that publishers will send them pre-release copies of games because they are on Metacritic.

    lowlylowlycook on
    steam_sig.png
    (Please do not gift. My game bank is already full.)
  • baudattitudebaudattitude Registered User regular
    So, considering how bonuses and occasionally studio closings are based on metacritic scores, a handful of review sites are now officially in the "kingmaker" seat.

    This can't possibly end well.

    TaranisurahonkyElvenshae
  • The_ScarabThe_Scarab Registered User regular
    So talking to a statistician I know, it seems like might not be as insidious as it first seems if the weighting corresponds to the number of reviews put out by each site as well as a few other things. I know that a lot of people at GAF seem shocked that GiantBomb is so low, but they have a small team and don't review every game released. Only something like 60 per year. Compared to IGN which literally reviews everything, hundreds and hundreds per year.

    I guess they're also weighted low because of their scale being five point rather than the normal hundred/twenty/ten scale. Less information, less distinction between titles.

  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    The numbers alone are completely meaningless to me personally. Without context as to how those numbers are arrived at, it's just noisy data. Why is Dark Zero (a site I've never heard of) considered more "weighty" than say, Destructoid? I have no idea, and without that information, it's just random numbers to me.

    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
  • The_ScarabThe_Scarab Registered User regular
    Of course, it could simply be all bought and paid for. The difference between the Official Magazines is notable.

    At the very least, this spells a big shakeup because which publisher is going to spend money courting a 0.5 site instead of a 1.5?

    BlendtecElvenshae
  • CadeCade Eppur si muove.Registered User regular
    Oh it definitely won't end well.

    The whole list is.....nuts. I can't see a lot of the sites that do reviews being very happy and those that are listed at the top are bound to be highly sought after by publishers while others feel like they've been jerked around until now, unless they knew which is definitely possible.

  • lowlylowlycooklowlylowlycook Registered User regular
    I really wish we could at least get that list with each site's relative traffic.

    And I wonder how often are these rankings changed. I have a feeling that FiringSquad isn't as big as it used to be.

    steam_sig.png
    (Please do not gift. My game bank is already full.)
  • lowlylowlycooklowlylowlycook Registered User regular
    From gaf:
    Wow O_O I write for DarkZero, which is a volunteer based site (we all have other jobs or are students, etc.)

    I have no idea how we are fixed into the highest category. O_O I'm shocked. We don't get paid or anything, we're just a group of guys who like gaming. O_O

    steam_sig.png
    (Please do not gift. My game bank is already full.)
  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    From gaf:
    Wow O_O I write for DarkZero, which is a volunteer based site (we all have other jobs or are students, etc.)

    I have no idea how we are fixed into the highest category. O_O I'm shocked. We don't get paid or anything, we're just a group of guys who like gaming. O_O

    I hadn't even heard of Dark Zero before this list was published. It also makes me wonder if there is anything nefarious at all here. If money was changing hands to get these ratings, I think the list would look much different.

    This again rolls me back to my original point: Without knowing how these ratings are arrived at, it's just noise. It gives us an idea of how Metacritic scores are calculated, but not much else. We also have no idea how fluid the list is.

    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
    Axen
  • mere_immortalmere_immortal So tasty!Registered User regular
    I don't know about you guys but the New York Times is the first place I go for all my reviews.

    Steam: mere_immortal - PSN: mere_immortal - XBL: lego pencil - Wii U: mimmortal - 3DS: 1521-7234-1642 - Bordgamegeek: mere_immortal
  • CadeCade Eppur si muove.Registered User regular
    One nice thing is that Polygon isn't listed. They are terrible as the Simcity scoring revealed and that was after some of their other questionable scorings.

  • AllforceAllforce Registered User regular
    Metacritic is now disavowing this list, of course without offering up any concrete info to refute it so...

    https://www.facebook.com/Metacritic/posts/501424766586647

  • The_ScarabThe_Scarab Registered User regular
    I don't know about you guys but the New York Times is the first place I go for all my reviews.

    To be fair, more people in the world will get their game reviews from NYT than from any hardcore gaming site, like Giant Bomb or Gamespot.

    Metacritic isn't just about games. It has a very broad demographic.

    I'm just sayin'.

  • AxenAxen My avatar is Excalibur. Yes, the sword.Registered User regular
    edited March 2013
    GnomeTank wrote: »
    From gaf:
    Wow O_O I write for DarkZero, which is a volunteer based site (we all have other jobs or are students, etc.)

    I have no idea how we are fixed into the highest category. O_O I'm shocked. We don't get paid or anything, we're just a group of guys who like gaming. O_O

    I hadn't even heard of Dark Zero before this list was published. It also makes me wonder if there is anything nefarious at all here. If money was changing hands to get these ratings, I think the list would look much different.

    This again rolls me back to my original point: Without knowing how these ratings are arrived at, it's just noise. It gives us an idea of how Metacritic scores are calculated, but not much else. We also have no idea how fluid the list is.

    Yeah I'm with you on this.

    The idea of weighted scores doesn't exactly shock me or outrage me. If anything it makes sense (depending on how the weighting is done). I mean, IMHO, it'd make sense that the review/score from a "respected" source would carry more weight then the review of some random dude from some random site.

    However, obviously that isn't how they have determined the "weight" of each site. Since I too have never, ever, heard of Dark Zero and if Domstercool is to be believed (which I see no reason to doubt him) it is a site comprised of random Joes who like video games.

    Although to be fair, those are the kind of opinions I prefer. :P

    edit- And I don't mean to single out Dark Zero. There are a fair number of sites on that list that I've never heard of who rank higher then other sites I'd describe as "household names" among the gaming community.

    Axen on
    A Capellan's favorite sheath for any blade is your back.
    GnomeTank
  • LuxLux Registered User regular
    I wonder if rating scale has anything to do with it. Because Giant Bomb's 5 star system doesn't and shouldn't translate to Metacritics 100 point value rating where a 3 is 60 points and a 4 is 80 points.

    ShadowfireDark Raven X
  • mere_immortalmere_immortal So tasty!Registered User regular
    edited March 2013
    The_Scarab wrote: »
    I don't know about you guys but the New York Times is the first place I go for all my reviews.

    To be fair, more people in the world will get their game reviews from NYT than from any hardcore gaming site, like Giant Bomb or Gamespot.

    Metacritic isn't just about games. It has a very broad demographic.

    I'm just sayin'.

    Yeah I get that exposure is a big factor here, with something like Yahoo games in the top tier too, still funny though.

    What's weirder though is the sites in the top tier I have never heard of. Dark Zero, Got Next? Maybe I stick to my internet comfort zone a bit much but no idea who they are.

    mere_immortal on
    Steam: mere_immortal - PSN: mere_immortal - XBL: lego pencil - Wii U: mimmortal - 3DS: 1521-7234-1642 - Bordgamegeek: mere_immortal
  • AllforceAllforce Registered User regular
    I think the point is less about Metacritic weighting scores and more about certain sites being seen as a better place to "invest" by publishers to insure a high score for their game.

    urahonkyElvenshae
  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    Axen wrote: »
    GnomeTank wrote: »
    From gaf:
    Wow O_O I write for DarkZero, which is a volunteer based site (we all have other jobs or are students, etc.)

    I have no idea how we are fixed into the highest category. O_O I'm shocked. We don't get paid or anything, we're just a group of guys who like gaming. O_O

    I hadn't even heard of Dark Zero before this list was published. It also makes me wonder if there is anything nefarious at all here. If money was changing hands to get these ratings, I think the list would look much different.

    This again rolls me back to my original point: Without knowing how these ratings are arrived at, it's just noise. It gives us an idea of how Metacritic scores are calculated, but not much else. We also have no idea how fluid the list is.

    Yeah I'm with you on this.

    The idea of weighted scores doesn't exactly shock me or outrage me. If anything it makes sense (depending on how the weighting is done). I mean, IMHO, it'd make sense that the review/score from a "respected" source would carry more weight then the review of some random dude from some random site.

    However, obviously that isn't how they have determined the "weight" of each site. Since I too have never, ever, heard of Dark Zero and if Domstercool is to be believed (which I see no reason to doubt him) it is a site comprised of random Joes who like video games.

    Although to be fair, those are the kind of opinions I prefer. :P

    edit- And I don't mean to single out Dark Zero. There are a fair number of sites on that list that I've never heard of who rank higher then other sites I'd describe as "household names" among the gaming community.

    Yeah, I'm not trying to single out Dark Zero, it was just the first on the high list, and jumped out at me as "Who?".

    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
  • The_ScarabThe_Scarab Registered User regular
    Allforce wrote: »
    I think the point is less about Metacritic weighting scores and more about certain sites being seen as a better place to "invest" by publishers to insure a high score for their game.

    Exactly this.

    Issue 1: The list is in the open. The weighting of scores is essentially going to become a weighting of marketing dollars put towards each publication. This is a roadmap for PR.

    Issue 2: The seemingly nonsensical weighting differences between strange publications.


    The first issue is much, much more meaningful and important. And more damaging to Metacritic. The second may well turn out to be a non-issue entirely. We don't yet know the way it is calculated (and it could be completely reasonable).

    Elvenshae
  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    To be fair, according to Metacritic, it's wrong on several fronts: They don't have that many tiers, the differences between tiers is not as great, and some publications are not only wrong on the list, but flat missing.

    The first two are the important ones to me. If Metacritic is internally using different tiers, and those tiers have a much smaller weight delta, then the data is completely, 100%, worthless, and if PR companies use it, they will fuck themselves.

    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
  • CadeCade Eppur si muove.Registered User regular
    GnomeTank wrote: »
    To be fair, according to Metacritic, it's wrong on several fronts: They don't have that many tiers, the differences between tiers is not as great, and some publications are not only wrong on the list, but flat missing.

    The first two are the important ones to me. If Metacritic is internally using different tiers, and those tiers have a much smaller weight delta, then the data is completely, 100%, worthless, and if PR companies use it, they will fuck themselves.

    On the other hand of course Metacritic will say it's wrong.....but where's the proof? They can show what they "use" but until the they show it we got no reason to believe the list is wrong.

    darklite_xElvenshae
  • AxenAxen My avatar is Excalibur. Yes, the sword.Registered User regular
    edited March 2013
    @GnomeTank

    Indeed.

    The_Scarab wrote: »
    Allforce wrote: »
    I think the point is less about Metacritic weighting scores and more about certain sites being seen as a better place to "invest" by publishers to insure a high score for their game.

    Exactly this.

    Issue 1: The list is in the open. The weighting of scores is essentially going to become a weighting of marketing dollars put towards each publication. This is a roadmap for PR.

    Issue 2: The seemingly nonsensical weighting differences between strange publications.


    The first issue is much, much more meaningful and important. And more damaging to Metacritic. The second may well turn out to be a non-issue entirely. We don't yet know the way it is calculated (and it could be completely reasonable).

    I'm not exactly a fan of Metacritic myself, for various reason we've gone over at length on this forum, so if anything I guess I hope that this destroys Metacritic's credibility and people's jobs/bonuses are no longer on the line because a reviewer was ordered to review a game from a genre he wasn't a fan of.

    Though I suppose that is wishful thinking. :(


    @Cade

    It is catch-22. If they do publish the list to show that the other one is wrong, then we are back to Issue 1. If they don't then Issue 1 is still a problem, only more harmful to all involved I guess.

    Axen on
    A Capellan's favorite sheath for any blade is your back.
  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    edited March 2013
    Cade wrote: »
    GnomeTank wrote: »
    To be fair, according to Metacritic, it's wrong on several fronts: They don't have that many tiers, the differences between tiers is not as great, and some publications are not only wrong on the list, but flat missing.

    The first two are the important ones to me. If Metacritic is internally using different tiers, and those tiers have a much smaller weight delta, then the data is completely, 100%, worthless, and if PR companies use it, they will fuck themselves.

    On the other hand of course Metacritic will say it's wrong.....but where's the proof? They can show what they "use" but until the they show it we got no reason to believe the list is wrong.

    You also have no reason to believe it's right. Grain of salt and all that.

    It's also not Metacritics responsibility to prove some researcher wrong, not in the slightest. They aren't under some order to give us the data, and they likely never will (for the exact reasons that Scarab pointed out, because it would open a huge can of worms).

    GnomeTank on
    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
  • AllforceAllforce Registered User regular
    And while Metacritic's self-described "tiers" may be smaller in discrepancy, it still comes down to statisticians running numbers to come up with what sites are given a higher weight.

    I'm guessing no matter what Metacritic says to the contrary their actual list of tiered sites aren't that far off from what this list actually is, since it's all computed from what I can only imagine is roughly the same pool of data.

    Elvenshae
  • The_ScarabThe_Scarab Registered User regular
    As we speak, statistics majors all around the country are working on confirming this data with a much larger sample size. Never underestimate how much fun it can be to be part of a gotcha moment.

    If this was wrong, as in not even close to being true, Metacritic wouldn't have commented. Why would they? Dismiss it out of hand. I used to work in PR and I would always tell clients that you don't deny something a little, you deny everything unequivocally or you don't say anything at all. A half-denial only breeds suspicion and investigation.

    Elvenshae
  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    edited March 2013
    The_Scarab wrote: »
    As we speak, statistics majors all around the country are working on confirming this data with a much larger sample size. Never underestimate how much fun it can be to be part of a gotcha moment.

    If this was wrong, as in not even close to being true, Metacritic wouldn't have commented. Why would they? Dismiss it out of hand. I used to work in PR and I would always tell clients that you don't deny something a little, you deny everything unequivocally or you don't say anything at all. A half-denial only breeds suspicion and investigation.

    They did deny it unequivocally. Did you read the post? It boils down to "he wrong, don't listen to him".

    Again, people seem to think that Metacritic is under some moral obligation to give us the data, if for no other reason than to "prove the guy wrong". Zzzttt, no.

    I am also curious what "gotcha" moment there is here. That Metacritic uses tiered weights? They've never denied it. Where is the gotcha? What lie is Matacritic being caught in?

    GnomeTank on
    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
  • reVersereVerse Attack and Dethrone God Registered User regular
    Cade wrote: »
    One nice thing is that Polygon isn't listed. They are terrible as the Simcity scoring revealed and that was after some of their other questionable scorings.

    The reason they aren't listed is because the study to aggregate the weighing system started before Polygon was a thing / had enough reviews to qualify / whatever technicality. Polygon is a Metacritic approved whatever and whatnot.

  • The_ScarabThe_Scarab Registered User regular
    GnomeTank wrote: »
    The_Scarab wrote: »
    As we speak, statistics majors all around the country are working on confirming this data with a much larger sample size. Never underestimate how much fun it can be to be part of a gotcha moment.

    If this was wrong, as in not even close to being true, Metacritic wouldn't have commented. Why would they? Dismiss it out of hand. I used to work in PR and I would always tell clients that you don't deny something a little, you deny everything unequivocally or you don't say anything at all. A half-denial only breeds suspicion and investigation.

    They did deny it unequivocally. Did you read the post? It boils down to "he wrong, don't listen to him".

    Again, people seem to think that Metacritic is under some moral obligation to give us the data, if for no other reason than to "prove the guy wrong". Zzzttt, no.

    I am also curious what "gotcha" moment there is here. That Metacritic uses tiered weights? They've never denied it. Where is the gotcha? What lie is Matacritic being caught in?

    The granularity of their data. Their slogan is 'Keeping score of entertainment'. If that score is reached on the back of a napkin, it severely damages their reputation. The seeming nonsensical nature of the weighting implies no cohesive method.

    Furthermore, their statement uses clever vagaries to sidestep the issue. Of course they're not going to release their actual method as negative proof, but they offer no conclusive statement on the overall ethics of a hidden weighting system. Which was the point of the research talk in the first place. People's jobs are on the line and Metacritic is needlessly secretive on that fact.

    Speculation is moot, however. Maths will win the day. Eventually. If a lie forces transparency for a poisonous industry juggernaut, then so be it.

    Elvenshae
  • lowlylowlycooklowlylowlycook Registered User regular
    GnomeTank wrote: »
    The_Scarab wrote: »
    As we speak, statistics majors all around the country are working on confirming this data with a much larger sample size. Never underestimate how much fun it can be to be part of a gotcha moment.

    If this was wrong, as in not even close to being true, Metacritic wouldn't have commented. Why would they? Dismiss it out of hand. I used to work in PR and I would always tell clients that you don't deny something a little, you deny everything unequivocally or you don't say anything at all. A half-denial only breeds suspicion and investigation.

    They did deny it unequivocally. Did you read the post? It boils down to "he wrong, don't listen to him".

    Again, people seem to think that Metacritic is under some moral obligation to give us the data, if for no other reason than to "prove the guy wrong". Zzzttt, no.

    I am also curious what "gotcha" moment there is here. That Metacritic uses tiered weights? They've never denied it. Where is the gotcha? What lie is Matacritic being caught in?

    Well the gotcha is that the weights make absolutely no sense.

    But that is also why I kind of believe them when they say that this list is all wrong.

    steam_sig.png
    (Please do not gift. My game bank is already full.)
  • OptyOpty Registered User regular
    The problem with trying to come up with a function to match an outcome is that there's infinite functions that can potentially create what you're looking for. As such--especially if you assume what Metacritic's said about their tiers to be true--it's highly likely the list is wholly incorrect. A nothing site can end up high on the list in such a situation because that's the only way their assumptions (how many tiers and the weight for each tier) coupled with the known data would fit the known outcome. Change the number of tiers used in the calculations to 3 and the weight for each tier to 1.1, 1.0, and .9 respectively and you'll come up with a different list.

    This is basically a nonstory unless you didn't know metacritic weighted review scores, then at least it informed you about that.

  • chocoboliciouschocobolicious Registered User regular
    Considering statistics is an entire branch of math dedicated to having an outcome and building models to support it, it is entirely possible to see the raw data and work your way backwards.

    That is, in fact, how the entire branch of all human mathematics and science was created.

    This isn't some infinite pandoras box with an impossible imaginary number. This is like, math, with a couple hundred thousand sources and a reasonably small number of pool entries.

    There isn't infinite functions because there aren't infinite entries. Your entries are finite both in the numerical data and the pool the data is being pulled from (Review scores and reviewers.)

    The only way you could literally not math this out is if they just rolled a die and reweighted every publication for every new entry for shits and giggles. Which is entirely possible as well.

    steam_sig.png
    BillmaanDrovekPixelated PixieElvenshae
  • ShadowfireShadowfire Vermont, in the middle of nowhereRegistered User regular
    Cade wrote: »
    GnomeTank wrote: »
    To be fair, according to Metacritic, it's wrong on several fronts: They don't have that many tiers, the differences between tiers is not as great, and some publications are not only wrong on the list, but flat missing.

    The first two are the important ones to me. If Metacritic is internally using different tiers, and those tiers have a much smaller weight delta, then the data is completely, 100%, worthless, and if PR companies use it, they will fuck themselves.

    On the other hand of course Metacritic will say it's wrong.....but where's the proof? They can show what they "use" but until the they show it we got no reason to believe the list is wrong.

    Until I see the long form weight chart, I don't believe them.

    WiiU: Windrunner ; Guild Wars 2: Shadowfire.3940 ; PSN: Bradcopter
  • StericaSterica Yes Registered User, Moderator mod
    Christ this industry is so fucked up.

    YL9WnCY.png
    Drunk_caterpillarCadeWybornAegeriLoveIsUnityTychoCelchuuuAxenAgahnimThe AnonymousChris FOMElvenshae
  • ShadowfireShadowfire Vermont, in the middle of nowhereRegistered User regular
    edited March 2013
    Cade wrote: »
    One nice thing is that Polygon isn't listed. They are terrible as the Simcity scoring revealed and that was after some of their other questionable scorings.

    I may not agree with some of their reviews, but I have tremendous respect for Polygon. They run some great long-form articles, and are pretty well written. And the change in SimCity's review score is fine by me... Call a turd a turd if it's true, and if the reality of a game after launch doesn't line up with what they saw beforehand, a change is warranted.

    Edit: I guess I just don't get the Polygon hate.

    Shadowfire on
    WiiU: Windrunner ; Guild Wars 2: Shadowfire.3940 ; PSN: Bradcopter
  • Drunk_caterpillarDrunk_caterpillar Registered User regular
    Rorus Raz wrote: »
    Christ this industry is so fucked up.

    It's a bummer to have to operate with a degree of cynicism at all times but let's be honest; every one of the cultural industries (film, TV, music, etc.) is at least this exploitative.

    Axen
  • ShadowfireShadowfire Vermont, in the middle of nowhereRegistered User regular
    Rorus Raz wrote: »
    Christ this industry is so fucked up.

    I guess I also don't see a huge deal here. There had to be a weight system in place for aggregating review scores, and they've admitted as much before. We don't know why different sites receive the weights they do, but it could be nothing shady at all, more based on their review score setup and how it meshes with metacritic's aggregate scores. Otherwise, Giant Bomb would throw review averages off considerably with their 5 point scale.

    I mean, metacritic is still terrible, and so that is definitely a "this fucking industry" thing, but this reveal, if it is one (it probably is) probably isn't a huge deal.

    WiiU: Windrunner ; Guild Wars 2: Shadowfire.3940 ; PSN: Bradcopter
  • OptyOpty Registered User regular
    Considering statistics is an entire branch of math dedicated to having an outcome and building models to support it, it is entirely possible to see the raw data and work your way backwards.

    That is, in fact, how the entire branch of all human mathematics and science was created.

    This isn't some infinite pandoras box with an impossible imaginary number. This is like, math, with a couple hundred thousand sources and a reasonably small number of pool entries.

    There isn't infinite functions because there aren't infinite entries. Your entries are finite both in the numerical data and the pool the data is being pulled from (Review scores and reviewers.)

    The only way you could literally not math this out is if they just rolled a die and reweighted every publication for every new entry for shits and giggles. Which is entirely possible as well.

    The structure and weighting of the buckets are a form of infinity in this case. Maybe the buckets aren't universal but are different by genre or something else? Maybe the buckets are universal but the weights are applied differently based on genre? Maybe there are no buckets and each site is given an individual weight? What if the weights change over time as a site becomes more popular or puts out more reviews? Are the scores locked in for a publication such that if their weight changes their old scores are still weighed by the previous weight or does their new weight retroactively apply?

    Basically there's too many possibilities left open in the data we have to reliably reverse engineer their weighting system. If you assume Metacritic isn't lying when they say the bucket/weight structure these guys used was incorrect, then the list is invalid.

  • AthenorAthenor Battle Hardened Optimist The Skies of HiigaraRegistered User regular
    What bugs me is a site like IGN getting 3 highest tier reviews, all treated as individual entities when there is likely an editorial oversight going on to make sure reviews come out similarly. But I have a long-standing grudge with IGN, so don't mind me.

    The thing about having different tiers or less tiers or less weighting really don't mean a ton in the end because the sample they tested from came to these conclusions.. which means their model is similar enough to the real thing that it can be used to game the system. I'm also willing to bet the publishers have access to the real numbers, or at least the big guys do.

    He/Him | "A boat is always safest in the harbor, but that’s not why we build boats." | "If you run, you gain one. If you move forward, you gain two." - Suletta Mercury, G-Witch
  • AegeriAegeri Tiny wee bacteriums Plateau of LengRegistered User regular
    edited March 2013
    GnomeTank wrote: »
    The_Scarab wrote: »
    As we speak, statistics majors all around the country are working on confirming this data with a much larger sample size. Never underestimate how much fun it can be to be part of a gotcha moment.

    If this was wrong, as in not even close to being true, Metacritic wouldn't have commented. Why would they? Dismiss it out of hand. I used to work in PR and I would always tell clients that you don't deny something a little, you deny everything unequivocally or you don't say anything at all. A half-denial only breeds suspicion and investigation.

    They did deny it unequivocally. Did you read the post? It boils down to "he wrong, don't listen to him".

    Again, people seem to think that Metacritic is under some moral obligation to give us the data, if for no other reason than to "prove the guy wrong". Zzzttt, no.

    I am also curious what "gotcha" moment there is here. That Metacritic uses tiered weights? They've never denied it. Where is the gotcha? What lie is Matacritic being caught in?

    The fact is Metacritic may as well have made an open challenge to hundreds of incredibly smart people, who will read the original paper (I assume that is somewhere?) and then confirm or deny what the other researchers found. While it is true the weights might be different, we'll have to see what the methodology of the paper in question is and how others react to it. Ultimately it doesn't matter jack shit what Metacritic says: It's going to be if other people can replicate the weightings that the original researchers found (which is how any science is done really). If Metacritic refuse to release their own numbers, they will have to live with what other people "confirm" to be their ratings numbers - because those doing this work will be the only ones providing evidence.

    So my question is the talk anywhere? What was their method? How well did their method apply to X scores? Is any of this available?

    Edit: Also, I may be contrary but I don't see anything wrong with a weighting system. How do you determine the value of a site that rates in 5 stars (as an example) vs. one with a 10 point scale? A 5 star review = 100 while getting a perfect 10 (even from IGN) is not anywhere near as easy. So you need some way to adjust these. I think their weighting is really bad if the researchers are correct, but a much smaller rating scale could account for that and even out the results.

    Aegeri on
    The Roleplayer's Guild: My blog for roleplaying games, advice and adventuring.
  • The_ScarabThe_Scarab Registered User regular
    If your weighting system is so complex that it can't be reverse engineered then I don't want anything to do with it. Transparency is the key here and the issue the original research was raising. People's jobs are on the line and Metacritic's weighting system, fundamental to their scores, is so obfuscated and changed so often for completely secretive that they laugh and sneer at people's attempts to decipher it.

    If Metacritic's weighting is so subjective, why do they exist at all? The whole point of a meta-critic is to attempt to reduce subjectivity by applying the rules of crowd opinion. Now it turns out their weightings are periodically 'adjusted' depending on a publication's overall quality. So now the meta-critics are critiquing the critics themselves. We're through the fucking looking glass here, goddam.

    AegeriAllforceplufimTychoCelchuuuTaranisurahonkyElvenshae
Sign In or Register to comment.