With the world and his dog keenly watching Metacritic, especially on the eve of big platform exclusives like our Killzone 2 and the 360’s Halo Wars, we caught up with Metacritic’s Marc Doyle and discussed scores, fanboys and big games. Great for your lunchtime read, we’re sure you’ll agree.
TheSixthAxis: Hi Marc. So, straight in: why do you think there is so much scrutiny around Killzone 2’s Metacritic score, more so than other PS3 exclusives like Uncharted or LBP?
Marc Doyle: I don’t think there has been any special scrutiny around KZ2 relative to other PS3 exclusives. It’s a highly anticipated game, and all highly-anticipated games create interest around their Metascores.
TSA: So is it just the fanboys that care about the Metacritic score with a Triple-A game so they can use it as ammo against the other side?
MD: I wouldn’t say so.
TSA: Is there any correlation between a Metacritic score and sales figures? Anything concrete where you can say “If a game gets 80 it will sell 2 million, 85 will sell 3 million”?
MD: Only anecdotally. Publishers have conducted their own studies using our data and have come to their own conclusions with respect to certain genres, but no, I’ve never conducted such studies.
TSA: But you’ve seen thousands of scores over the years – how accurate are you at predicting what a new game’s Metacritic score will be?
MD: Not very. After seeing the first 5 or 6 reviews, though, I’m pretty good at predicting generally where the final score will be.
TSA: But do you find that you can predict certain publications / sites scores with any accuracy?
MD: Again, if I’ve seen a pretty sizeable group of reviews, I often suprise myself at my ability to predict what certain publications will award the game.
TSA: Which of your personal favourite games has ended up with a disastrous Metacritic score?
MD: I don’t own any current or last gen consoles. I was a pretty avid gamer during the Genesis [Megadrive] days, especially when I had a year off between college and grad school, but since then, I haven’t had the time to devote a huge slice of time to gaming. Too busy generating Metascores!
TSA: Have you ever been sent any interesting emails asking you to massage the scores for certain games?
MD: No. I’ve been asked why we include certain publications and why we don’t include others, and I’ll get complaints that the critic giving the lowest score to a popular game should be excluded, but that’s about it. I’ve never been asked by my parent company or publishers or developers to alter my process and, by doing so, “massage” a score as you say.
TSA: The press is starting to run stories about developer’s bonuses being linked to Metacritic scores. How do you feel about that given that your choices – which publication’s reviews to accept, for example – may ultimately affect that?
MD: It doesn’t affect my decision-making with respect to what publications I choose or how I weight them in the least. My mission has always been to educate my users – buyers and renters of videogames – with respect to which games are worthy of their valuable entertainment dollars.
TSA: But do you think it’s a fair way of compensating developers?
MD: I’ve never seen an actual clause in a contract which uses Metascore as a threshold for a bonus or penalty, so I can’t comment on it in any educated way. In general, I think it’s a good thing that publishers are striving for quality in addition to sales. But I think they should go about striving for quality in a fair-minded way. But again, every contract is different.
TSA: Other people seem to favour the tried and tested method of using sales success as a benchmark. However, I think that can be misleading because we don’t like every game we buy and I’m sure that’s never communicated to the publishers on a big scale – so is Metacritic the best judge of a game’s quality?
MD: Is it the “best” judge? I can’t say that for everyone in every capacity. It’s one way to judge success.
TSA: So to anyone looking to get their site on Metacritic, or are interested in the selection process, what’s the process a site / publication must go through in order for you to start including their reviews as part of the Metacritic score?
MD: I’m constantly evaluating new sites that spring up thorughout the year. Either I’ll contact them or they’ll contact me, but I generally send a questionnaire to get to know their operations a bit better. Then I read a bunch of their reviews, focusing on the quality of their analysis, their scoring integrity, and whether they’re able to support their scores with reasoned and well-considered arguments.
I also want to make sure that the publication has established a reputation for credibility and reliability. I rarely will pick up a new site because such a reputation takes time to establish, but if a new site springs up with established critics who i know and who I’ve evaluated in the past, I can make an exception. I don’t have specific number thresholds for reviews when deciding which publications to track.
TSA: TheSixthAxis, already listed on Metacritic, is a PS3 site, so which PS3 games have had Metacritic scores that surprised you, good or bad?
MD: Nothing surprises me about scores.
TSA: If you were reviewing Metacritic what would the review say, and what would it score?
MD: Plenty of people, from journalists to regular users, have reviewed Metacritic, and many have done a very good job, pointing out our strengths and weaknesses. Many of their observations have helped us improve our offerings. It’s not my place to review the site or to give it a score. What would you give us?
TSA: Ah, that would be telling. How’s your multiplayer?
And on that 5/5 bombshell, there ends the interview. Thanks to Marc for taking the time out of his very busy schedule to speak to us, and our very own Michael for the questions. Good luck with the site, Marc.