Sunday Thoughts: Reviewing Reviews
Head banging.Published: 16:30, 05/09/2010 by Staff.
12When you post something on the internet, you’re openly inviting discussion and critique. We’re lucky here at TheSixthAxis because we’ve managed to create and nurture an active community and a readership that’s happy to voice their thoughts openly and without – wherever possible – fear of any sort of backlash or abuse – which is great we love reading your comments and most of the time they really make our day, but the one aspect that still confuses us on a regular basis is how people react to reviews.
Our review policy is linked to at the foot of every page, but it still appears that some of our readers don’t really understand what our reviews are all about, quickly flicking to the numerical score at the foot of the text and deciding that the game is, for example, ‘average’ because it’s a 7/10 score. 7/10, as you’ll see from our review policy, is ‘quite good’ – inFamous and Heavy Rain are, in my personal opinion, quite good. That’s just my opinion, yours may differ and that’s fine, that’s human nature. By that’s the review score we went with and despite the comments at the time, I still stick with both those scores.
Average, though, by definition, is very much 5/10.
Games that we’ve ranked as ‘average’ recently, then, include Bladestorm, Haze and Enemy Territory: Quake Wars, and, as of this weekend, Mafia II. Metacritic says we score games, on average, 1.8/100 points higher than the industry average, a point which reader Jacklum raised earlier today in the comments of the Metroid review. This, in my opinion, is completely irrelevant and as our editor Peter said with regards to his Mafia II review, we don’t get caught up in what other reviewers are saying about games in any way – it doesn’t matter.
In fact, knowing what others were scoring a game in advance of a title we were reviewing might even be counterproductive. We try, as much as possible, to ‘go in blank’.
What’s perhaps more alarming is the way the industry outside of TheSixthAxis (and a couple of other publications) has already set a ’7′ as the ‘average’. One reader of the site said that “actually a 7 is a average score, 5 would be the middle of the score-scope, so to say, but the way games are scored, 7 is the average. I dont say that’s the best way to do it, i also would prefer if 5 would mean average, but once a standard is settled, its hard to change.” It’s not hard, at all, it’s easy – don’t treat a 5/10 as anything but average.
And, for goodness’ sake, don’t think a ’10′ is perfect.
The way I see it it’s others that have skewed the score upwards. I don’t know why or when this happened and any attempt to try to work it out is well beyond the scope of this blog – besides, I’m not privy to the editorial policies of any other site apart from ours. But I do know what when we review a game we try to ensure that, if nothing else, that middle point is always at the forefront of our minds. And it’s the text that takes us days to produce, not the number at the bottom that, as much as a single integer can do, is meant to summise everything that’s above it yet provokes so much confusion.
We’ve blogged about review scores here on TSA before, but it’s worth re-discussing perhaps, especially as it can be so disparaging going over old ground in the comments for every new review. Essentially, we want you to read the text, weigh up the pros and cons and then, if you desire, glance at the score at the botom but always take that score in the context it’s intended. All reviews are subjective to some degree, that’s personal opinion for you, but we do balance out the good and bad as objectively as anyone can do.
And, as we’ve said before, look at as many other reviews as you can do before making up your mind.
Disclaimer: All comments are the opinion and responsibility of the individual author and not TheSixthAxis. You must read and agree to our terms before reading and commenting on this site. User comments are not always moderated by TheSixthAxis.
« Previous 1 2 3
« Previous 1 2 3
Most Comments This Week
- PS3 Firmware 4.10 Brings, Wait For It, A Decent Browser! [101]
- OnLive’s Evolution [72]
- CoD: Black Ops Has The Best Ending Ever [67]
- PS Vita TV Ad Campaign Revealed [65]
- PS Vita’s Load Times Analysed [56]
- What We Played #39 [49]
- PS Vita: Relic or Revolution? [47]
- PSN Store Update: 08/02/12 [45]
- Vita PSN Prices Announced [44]
- Is The PS Vita’s Marketing Really On Target? [43]
TSAtv: Original Video Content
Batman book review
Kris Dancing, again
Joe Danger interview
Sonic Generations
05/09/2010 at 22:25
Member since: Jan 2010
If you’re going to go against what has, by your own admission, become industry standard, why not leave the score off completely?
To me, the score is only useful to get an idea how a game is being received across a range of reviews. As good as you think your reviews might be, I’m not naive enough to read one single review and assume it to be definitive.
There is a fantastic tool called metacritic, where gamers can quickly gauge a game’s reception with a quick scan of score and summarising comments. All TSA’s scoring methodology does is bring down the average (and thus perception) of otherwise great games. As I said above, if you don’t agree with the industry standard score average I’d rather you didn’t score at all.
05/09/2010 at 22:42
Member since: Forever
What delicious irony.
In an article that is primarily about asking our readers to read the copy we publish you appear to have not spotted both the reference to Metacritic in the blog, and the line that shows we, on average, score higher than the average on the aggregator.
An aggregator that we wouldn’t be on if we didn’t have scores.
06/09/2010 at 09:50
Member since: Jan 2010
There is no need to be patronising, I read your article :P.
1.8 point on a 1-100 point scale is a completely insignificant figure. You score out of 10, and you’re within 0.18 of the average, so for all intents and purposes you score the same as everyone else. Turns out you’re not a delicate snowflake after all ;).
That tells me one of two things, either:
a) TSA actually _do_ think and score to the same standard as everyone else despite your protestations to the contrary…
or
b) You really _do_ set your scoring on a linear scale from 0-10, but evidently are more easily impressed than the majority of sites that unofficially score from 4-10 (with 7 being average).
Either your scores don’t match your convictions or on average you’re reviewing with slightly more rose coloured glasses than the other sites. I realise this doesn’t gel with what I said above about your lowering the average score (which on reflection doesn’t make sense given your metacritic average), but if you were actually using your prescribed scoring methodology it would.
Not that it matters, as I’ve alluded to your reviews would be fine without scores. Generally speaking they are thorough and cover the areas that are of most interest to your reader base. I still don’t see the point of getting on the soap box about 5 being average when you clearly don’t actually score that way anyway (on average). Sure for Mafia II you did, but reading through metacritic I can show you a bunch of games TSA say are crap and still give 5/10 to, likewise several where the word picture screams ‘average’ but you’ve awarded 7.
All you’ve done with this article nofi is prove to me that reviewers everywhere, including here, are nothing if not inconsistent. The metacritic average remains a great way to sort the wheat from the chaff and get a pretty accurate ‘global’ score. As for TSA reviews, the words are great, the numbers are largely pointless.
These remarks are meant to be discouraging in anyway, if anything it’s refreshing to be a part of a site that is willing to discuss such things. Whenever I bring up something similar on other sites (such as ones that might rhyme with boy stick) my comments are removed and account banned!
06/09/2010 at 09:56
Member since: Jul 2009
Jesse – did you manage to catch the comments about how TSA have to cherry-pick (somewhat) the titles they review? The reason why the average scoring game is that much higher when looking at all titles here.
06/09/2010 at 10:11
Member since: Jan 2010
That’s irrelevant given how maths works.
They are 1.7% (ie: 3/5ths of sfa) above the average score for any given title. Therefore, for any given title – TSA score on average exactly the same as everyone else.
Of the 233 reviews they’ve done, over half (52%) were scored higher than the average critic, yet TSA claim* to have a more linear scoring methodology than most other website (*implicit in the context of this article). It doesn’t gel.
Their metacritic average is only against the titles they have reviewed, so even if they are a majority of quality ‘hand picked’ titles, they still score higher than most other sites. Other sites which, if assumptions are to be believed, have benchmarked their ‘average’ score higher than TSA.
Anyway, it’s been a long day and I didn’t mean to start a math war here. Just saying that if scoring is so contentious why not just ditch it? I’m sure TSA readers would understand.
06/09/2010 at 10:16
Member since: Jul 2009
It’s everything given how maths works.
If you’re trying to take the mean average of twenty numbers and (like TSA) you cherry pick the numbers then the average means something quite different to other sites. TSA has only so much time on its hands and actively tries to avoid real doozies unless they’re decent sized franchises that people expect to have reviewed (good or bad).
Once the figures are loaded like this, the average appears to fit in with the industry norm even though per title, it still doesn’t.
06/09/2010 at 10:59
Member since: Jan 2010
Again, no.
When metacritic states TSA’s deviance as 1.7% from the mean (compared to other critics) it is ONLY referring to the titles both TSA and the other critics have reviewed in common. In that context is makes NO difference whether the title was good, bad or ugly – as we’re only looking at the average deviance from the mean score awarded for each title.
To put it another way, TSA have review 233 titles (according to metacritic), where as IGN have reviewed reviewed 7,214. When metacritic calculates TSA’s average score deviation compared to other critics, they are only using data for the 233 titles TSA has registered a score for. They don’t care about the thousands of other titles that other sites may or may not have reviewed.
To that end, even if IGN had reviewed 5,000 doozies, they won’t be considered when TSA is compared against them… only the titles BOTH have reviewed are compared to calculate TSA’s deviance from the mean.
Your homework for tonight is all of chapter 10.
06/09/2010 at 11:48
Member since: Jul 2009
I didn’t realise Metacritic did that (with how it worked out the deviation). Cheers, matey.
07/09/2010 at 09:04
Member since: Oct 2008
@Jesse: Great, somebody with more nerve, time and better english than me. thanks for expertly eyplaining what i was trying to say all the time :)
05/09/2010 at 23:22
Member since: Jul 2009
TSA being stricter with scores gives every title a more balanced metacritic score (if you go down that route). Regardless, the community on TSA know that an 8 out of 10 is well deserved. A 9 is something special and a 10 is worth reincarnating your dead grandmother over.
“As I said above, if you don’t agree with the industry standard score average I’d rather you didn’t score at all.”
What the hell? Really! So if 15% of sites (for example) think the industry mean of 6.5 or so shouldn’t denote an average game they should avoid scoring altogether. What a load of claptrap.
06/09/2010 at 09:56
Member since: Jan 2010
I’m saying if everybody knows that the standard average game get’s 7, then one site bucking that trend isn’t really doing much other than confusing people who aren’t familiar with their particular scoring methodology.
If you want to make people actually read a review and come to their own conclusion, leave the score off! If on the other hand, you are going to use a score – at least make it consistent with what people are used to seeing. Either way, it doesn’t really affect _me_ at all, I always shop around for my reviews so outlying scores like TSA aren’t going to dramatically alter my opinion one way or the other. It’s the words that matter most to me.
06/09/2010 at 10:17
Member since: Jul 2009
Jesse – agree with you too, as a random aside, that it’s a great place to even discuss this without us smack-talking each other into oblivion.
06/09/2010 at 00:35
Member since: Aug 2009
“And it’s the text that takes us days to produce, not the number at the bottom that, as much as a single integer can do, is meant to summise everything that’s above it yet provokes so much confusion.”
Well, when you print inconsistent reviews like that Uncharted 2 one you did, the confusion is understandable. You spent the entire review crapping over the game and yet gave it 9/10?
I don’t judge games by numbers and agree with much of the sentiment this srticle displays, but there is a point to be made on occasions where the review text seems completely at odds with the score/summation given to a game that the quality of a review needs to be questioned.
06/09/2010 at 02:34
Member since: Aug 2008
What a fantastic article and amazing debate. It’s stuff like this that seperates this site from all the others. A well constructed argument and some fantastic non-fanboy, flame retardent commrents. Thats why we come back. I will now remove my head from up Nofi’s arse and state my point. If it causes so much crap why not hides the final score behind a spoiler/reveal thing. Oh and the first 1 to +1 will get abused!! (S-A-R-C-A-S_M)
06/09/2010 at 07:20
Member since: Jul 2009
+1! I want my abuse ;)
06/09/2010 at 08:38
Member since: Feb 2009
In the great old days of Sinclair User , Your Sinclair , Crash and C&VG the scores where judged as you wish they are now . It wasn’t uncommon for a game to get a 1 ,2 or 3 if it was pants but a game never gets that now .
Last Thursday i downloaded the Vanquish demo and strangely for a geek like me hadn’t heard anything about it . I was about to do a quick google and look on various forums for an opinion on whether it was worth downloading but then i thought sod it I will make my own unaffected opinion .
Well i loved it and am not generally a fan of 3rd person games I thought great I need to know more about this game so went a looking and found a lot of people saying it was not very good “a poor mans gears of war” was one that I remember . If id read these opinions before trying it for myself I doubt I would have bothered .
06/09/2010 at 10:16
Member since: May 2010
i love reading the reviews on here and you have to remember its all opnion so of course there is going to be a few games i dont agree with, eg ufc 2010 a 10, NO!
also i see 6-7 as a good score and higher a very good game. there are a few reviews on here which i think should be higher and a few that should be lower. but overall a review is a persons opnion and they cant be wrong.