Sunday Thoughts: Reviewing Reviews

TSA Staff 84 Comments

When you post something on the internet, you’re openly inviting discussion and critique.  We’re lucky here at TheSixthAxis because we’ve managed to create and nurture an active community and a readership that’s happy to voice their thoughts openly and without – wherever possible – fear of any sort of backlash or abuse – which is great we love reading your comments and most of the time they really make our day, but the one aspect that still confuses us on a regular basis is how people react to reviews.

Our review policy is linked to at the foot of every page, but it still appears that some of our readers don’t really understand what our reviews are all about, quickly flicking to the numerical score at the foot of the text and deciding that the game is, for example, ‘average’ because it’s a 7/10 score.  7/10, as you’ll see from our review policy, is ‘quite good’ – inFamous and Heavy Rain are, in my personal opinion, quite good.  That’s just my opinion, yours may differ and that’s fine, that’s human nature.  By that’s the review score we went with and despite the comments at the time, I still stick with both those scores.

Average, though, by definition, is very much 5/10.

Games that we’ve ranked as ‘average’ recently, then, include Bladestorm, Haze and Enemy Territory: Quake Wars, and, as of this weekend, Mafia II.  Metacritic says we score games, on average, 1.8/100 points higher than the industry average, a point which reader Jacklum raised earlier today in the comments of the Metroid review.  This, in my opinion, is completely irrelevant and as our editor Peter said with regards to his Mafia II review, we don’t get caught up in what other reviewers are saying about games in any way – it doesn’t matter.

In fact, knowing what others were scoring a game in advance of a title we were reviewing might even be counterproductive.  We try, as much as possible, to ‘go in blank’.

What’s perhaps more alarming is the way the industry outside of TheSixthAxis (and a couple of other publications) has already set a ‘7’ as the ‘average’.  One reader of the site said that “actually a 7 is a average score, 5 would be the middle of the score-scope, so to say, but the way games are scored, 7 is the average. I dont say that’s the best way to do it, i also would prefer if 5 would mean average, but once a standard is settled, its hard to change.”  It’s not hard, at all, it’s easy – don’t treat a 5/10 as anything but average.

And, for goodness’ sake, don’t think a ’10’ is perfect.

The way I see it it’s others that have skewed the score upwards.  I don’t know why or when this happened and any attempt to try to work it out is well beyond the scope of this blog – besides, I’m not privy to the editorial policies of any other site apart from ours.  But I do know what when we review a game we try to ensure that, if nothing else, that middle point is always at the forefront of our minds. And it’s the text that takes us days to produce, not the number at the bottom that, as much as a single integer can do, is meant to summise everything that’s above it yet provokes so much confusion.

We’ve blogged about review scores here on TSA before, but it’s worth re-discussing perhaps, especially as it can be so disparaging going over old ground in the comments for every new review.  Essentially, we want you to read the text, weigh up the pros and cons and then, if you desire, glance at the score at the botom but always take that score in the context it’s intended.  All reviews are subjective to some degree, that’s personal opinion for you, but we do balance out the good and bad as objectively as anyone can do.

And, as we’ve said before, look at as many other reviews as you can do before making up your mind.

Tags:

Related stories from TSA

Sunday Thoughts: Our Friend Peter
Sunday Thoughts: Control is Key
Sunday Thoughts: My Friend Kris
Sunday Thoughts: The Second Screen Solution

84 Comments

  1. I wish this topic would piss off. I had my say into that little debate about 10 mins ago.

    I wish people would learn to read rather than look at a bunch of numbers on metacritic.
    Why do we need scores anyway?

    • Yep do away with scores. There is not point them. If I was even slightly bothered about buying a game I’m going to read the whole review anyway regardless of what number is at the bottom.

      • The score is the second thing I look at. Its actually the pros and cons I read first. If they spike my interest then Il check the scored if it has a 7 or more il read the review.

    • Why do we need scores? It’s simple – people are lazy. They just want a numerical value to that will decide for them whether to pre-order, buy on release, wait for a reduction in price, rent or just avoid a game completely.

      I would bet that at least 50% of views on an article page are people who are interested solely on the single number at the bottom and not the content of review. Even I admit to doing this sometimes. When Joystiq decided to do away with review scores, it honestly made me less inclined to read the review.

      I think it’s because it’s very hard to tell whether a game is good or bad and how it compares to other games in its field by just reading a review. Words are very subjective whereas a number is simple and clear (although even THAT is pretty subjective within the world of gaming journalism).

  2. i think some people were once scared to give an average game 5/10, the trend is everywhere now though, so I would agree with nofi that 7/10 is the ‘accepted’ average….

    • but if /everyone/ gave an average game 5/10, there wouldn’t be a problem and people wouldn’t over-react to their perception of /our/ scores….

      • i think the problem is that you (TSA) mainly review good games so when an average one comes out that you review its all *shock horror* when they look at that little number at the bottom.

        *searches for Dragon Age:Origins* playing that now and really liking it although has a fair few flaws.

      • Just read DA:O review. Cb loves his RPGs it seems :)

        Do all articles get a comment expiry put on them after a certain length of time?

      • that would be the ideal case. but i dont think it will happen… more likely the scores will vanish…

      • What Roy said. It’s not like this site reviews all games ever, there’s more of an emphasis on the larger releases and better titles.
        So TSA’s average score (Arithmetic Mean) must be quite high, despite seeing 5/10 as the score of a middle of the road game.

        Course, nobody will ever be happy. You should see Edge-Online after yet another PS3 exclusive gets a 7/10. There is literally hell to pay for that. I think the biggest was when Gears 2 got 9/10, but then Resistance 2 got 7/10 and a few months later, so did KZ2…

        The only real way to escape this is to eschew review scores. Ars.technica doesn’t bother with them, instead going with a Buy, Rent, Avoid system…

  3. Haters gonna’ hate. Don’t change TSA, don’t change.

  4. I can honestly say that scores don’t make a great deal of difference to me these days, I’m much more interested in the general feelings and opinions of the reviewer though I feel it would be interesting (particularly on high-profile titles) to get a review opinion of three or four TSA writers as it may help from a balancing P.O.V.
    I understand it would be a lot of work but it’s maybe something to consider for GT5 or something for instance, you guys did something similar for MW2 which I found utterly fascinating even though I wasn’t particularly interested in the game itself.

    Great piece, nofi and keep up the good work chaps. :)

  5. After reading the review, I generally ignore the score and just look at the Pros/Cons. I think that’s one of the best things about the reviews on TSA, the Pros/Cons section will simply give you the facts.
    Ill always check Metacritic just to see what the range of scores are, but generally I judge whether to buy a game either on a Demo or gameplay videos.
    I think that if someone decides to buy or not to buy purely on a score out of 10, then they are stupid, and end up missing out on some really great games that only get a 5/10 which is deemed to be ‘crap’ by them.

    • I agree, if TSA review something I’m not that interested in, I’ll often have a quick look at the Pros and Cons. They are similar to Gamespot’s “awards” that show you the good and bad points. I wish more sites used the same kind of thing.

      A game I’m looking to buy gets a Metacritic search, then I’ll read reviews from publications that I trust, like TSA, Gamespot, IGN (awful site, but good revies :) )etc

  6. I would the “7/10 as average” grading standard is just a by-product of the educational grading system. In school, a 7/10 is a C grade…which is average.

  7. The way I personally see it is:
    10=Absolutely amazing, buy it now
    9=Well worth a buy, but not an instant classic
    8=Really quite good
    7=pretty good, if you’re a fan of the genre you’ll probably like it
    6=buy it if you want a game and there isn’t anything better out, but probably has a few flaws.
    5=not great, there are much better games out there
    4=worth a buy for a quick fix in the genre, but no real lasting appeal
    3=Nice idea, very poor execution
    2=Avoid
    1=utterly atrocious, don’t be tempted even if you see it for £3.

    I think the problem is that there are so many really good games, anything less than 70% is likely to indicate that there is something better out there

    • You think a game that scores 40% is worth a buy?

      • Reading it back, maybe not. I think I was thinking of if you really wanted a FPS, for example, to tide you over until an imminent big release, if you saw a 4/10 rated FPS for under a tenner (maybe £5 preowned), it might be alright.

  8. After reading that I still think 5/10 is average and 10 is perfect, TBH 7/10 I think is very good.

  9. seeing as i get quoted in this article, i somehow feel the urge to reply :)
    Let me start by saying i dont feel attacked or anything, to get that out of the way.
    I want to clarify that i did read the text of the review, and i didnt reall agree with it and i wrote why. i didnt say the review is wrong or bad, i just have a different opinion, and i didnt even mention the score in my original post.
    im writing this because i want to distance myself from “some of [the] readers [that] don’t really understand what our reviews are all about, quickly flicking to the numerical score at the foot of the text and deciding that the game is, for example, ‘average’ because it’s a 7/10 score”
    i dont really give much about the review score…

    the above quote came into existence, because somebody stated that the average score her on TSA is a 5, because the scoring policy, which is also mentioned in this article says so.
    But what the article doesnt mention is what i later wrote, that the average score that TSA gives in the reviews is a 7.5.
    And in my opinion the scores that are given on a daily basis are more important than whats stated in the policys…
    one could argue that the average given score is higher than 5 because the games are in general better than average, but thats a bit contradicting, if games are in general better than a set average, than this average is no longer the average and has to be adjusted…

    • Two different things.
      The arithmetic mean average is skewed because TSA doesn’t really review games that are complete garbage, instead tending to stick to the larger releases..
      So TSA average score is 7.5, but the score that TSA would give to an average game would be 5.

      It’s a subtle but important distinction.

      • I’m pretty pleased in general with the titles TSA review, particularly the small PSN based games, I can go anywhere for a general release review but the PSN reviews really give me valuable information and opinion.

        Obviously there’s always room for more but for the site that TSA is, I think it does a pretty sterling job.

    • “than this average is no longer the average and has to be adjusted…”

      an argument so fundamentally flawed I don’t know where to start.

      • why? all im saying is that the review scores (in general, im not talking about TSA here) have to be readjusted so that 5 is again the average. because now, 7 is about the average (again, not talking about TSA here…)
        so. wheres the problem?

    • Let me throw some maths at you. The 7.5 is a MEAN from all TSA’s reviews. It adds all the scores together and divides by how many reviews there are. This means that the MEAN average of all their scores is 7.5. That number is not what TSA consider as an AVERAGE score. The average that TSA use is the MEDIAN. This number is the very middle of the scale they use. In this case, it’s a 5. Half of the top score, 10.
      Think of it as a 3 point scale. The bottom (0) is the worst. In the MIDDLE (5) is AVERAGE. The top (10) is the best (or near perfect). Now using this 3 point scale, you can see that a 7 review is ABOVE average.
      If you still don’t understand, I drew a pretty picture to explain my (and TSA’s) point:

    • We don’t review every game, we have a skew towards downloadable titles and big hitters. This is mostly because we don’t get review code for everything, so we have to rely on staff deciding to spend their own money on some titles. Now given that it’s their own money staff are less likely to review games which have a pre-conception of being poor, based off of previews or previous entries into the series if the game is part of a franchise.

      Given these facts it seems that the mean of our games will be skewed higher, as we’re less likely to play stuff that’s truly junk. However the median of our review scores is 5, as we score using only the integers between 1 and 10 (technically the median would be 5 and 6, but we’ve decided on 5 as it’s simpler to only use one score as the median for our purposes).

      So yes, 7.5 may be the mean of the scores for games we play, but as we don’t play every game reviewed it’s not indicative of what we actually consider an average game from the set of all games ever released.

      • I’m just disappointed the review policy doesn’t allow for 0 out of 10. :-p

      • I think if you get 1 it basically means “Yes this was actually a game and not a block of cheese”

      • I’d prefer the block of cheese if the game only got a 1

  10. Um, I guess this was my fault then. I just felt that TSA was kind of looking down on ‘the industry outside of TheSixthAxis’ for setting the average too high when, in practice, TSA actually gives slightly above average scores than other sites. Seemed like a discrepancy to me.

    I personally don’t have a problem with the scores you award as, for the most part, the reviews are well written and this is obviously the most important factor.

    No hard feelings?

    • Not just you man. Most reviews get the same kinda comments. “that score is wrong” ” but you gave X game this and this is better” etc etc etc

      Pointless argument enducing most of the time.

Comments are now closed for this post.