In the first part of this series we looked at how the rush for traffic has tainted the reliability of many sources and how the internet is beating print media to the punch and therefore squeezing the profit out of printed news sources. The second part saw an examination of the relationship a journalist has to keep with his sources and how that might affect his judgement. In this third and final part I will be looking at a huge part of our industry and whether the system it requires is unworkable.

Reviews and scoring systems.

In recent years there seems to be a move away from the historical reliance of the game-buying public on reviews. I believe that this has happened because of the immediacy which the internet has provided us and the marketing opportunities that developers and publishers now exploit to the fullest.

The consumers are now far better informed about a game than they ever have been, long before it is ready for review. This leads onto one of the minor problems inherent in the reviewing system employed by many journalists in recent years; reviewing an early code of the game. Journalists struggle to get a review to their audience while it is still relevant because if your review isn’t out before the release date then the pre-orders and first day purchasers are already playing the game. So they get early code to review and can’t comment on the actual finished article.

When your review is being read by a huge number of people who made their decision based on gameplay videos and “hands on” previews they don’t want an unbiased, balanced opinion on the game. They want validation for the opinions they have already formed. You don’t want to spend your money on a product you believed would be great and then listen to someone criticise the game you’ve been persevering with. You want someone to tell you that it’s ok, that cash was well spent. You were right all along.

This is true of the consoles too. I spent £400 on my PS3, I don’t want some snarky Xbox owner telling me that the online service isn’t up to scratch, I want to believe that I’m getting great value for money. It works both ways too. An Xbox owner who spent their hard-earned cash on their console doesn’t want me sneering about red rings and extortionate add-ons that came in the box with my console.

So should the reviewers stick to their guns, view a game objectively and write about that experience or should they cater for their audience for the sake of popularity and continued employment? It’s hard to argue for integrity when you see that it also means unpopularity and that means unemployment.

So what is the solution that journalists came up with? The solution is skewed scoring systems.

For the sake of argument I’ll use an “out of ten” scoring system in this article simply because it is easy to see how that translates to the “5 stars” and “percentage” systems or even the “letter grade” system which has started popping up.

So what do I mean by skewed? Well, if a game is brilliant it gets a 9 or 10 (more on 10s in a moment) which is fair enough. Brilliant game equals high score. What if a title has been marketed extensively, raised a lot of excitement and then it gets an 8 in a review? It would be disappointing. A lot of us generally see an 8 as a poor score for a game with high expectations. If Killzone 2 had gotten 8s we would have been annoyed. What about a game which is really not very good? If you read a review score and it awards a game 6/10 is your first thought “above average” or is your first thought “absolute rubbish”?

Shuffling around Metacritic (more on this later too) I found games like “The Club” and “Conan” around the 70% mark. That’s 7/10 for games which are almost universally agreed to be awful. I’ve never met anyone that will admit to owning them but I have seen a number of copies gathering dust on the pre-owned shelves in Game. Further investigation shows that to be around the 40% mark which is just below average, you have to be as bad as “Go! Sports Skydiving” or “The Golden Compass” which are both just about unplayable. Nothing on Metacritic scores below a 3/10 for the PS3.

This is proof that, in general terms, reviewers don’t use the lowest third (1-3) of their scoring system. Add that to the fact that they use the highest third (8-10) of their scoring system over a third of the time (112 out of 316 Metacritic PS3 entries scored 78 or above) and you can see that the scoring system is skewed. A game has roughly a 33% chance of getting an 8,9 or 10, roughly a 66% chance of getting a score between 3 and 7 and no chance of scoring lower than a 3. Are there really no terrible games?

So even though the scoring system is generally skewed what about the games that get top marks, are they perfect? This is a misconception which is inevitable when you introduce a metric scoring system. If you go to school and get 10/10 in a test then you answered all the questions correctly, your test was perfect. Reviewing a game doesn’t work like that; they are marked against their competitors and against the reviewers opinion. We don’t get a list of checkboxes to mark a game’s performance.

Does GTAIV deserve its 10s? That’s a matter of opinion. It was certainly better than anything else at the time and it had fantastic production values so maybe. Was it perfect? Absolutely not but if you’re giving “Conan” a 7/10 then surely a game with the scope of GTAIV is at least 30% better? Surely though, “Conan” is 30% better than “Go! Sports Skydiving”? You should be starting to see the problem now. Although if so much of a review is based on the opinion of the reviewer why does the final score matter so much?

Recently we’ve heard unconfirmed (but heavily endorsed) reports that certain developers and publishers are now offering bonuses and contract-extensions based on Metacritic results. This essentially converts the artistic, creative pursuit of game-development into a metricated field where those who can pander to the opinions of a small group get the money and the ability to continue making games. It is an alarming trend which is, thankfully, being almost universally criticised.

So where do we go from here? Well, you are the only people who can make a change. Reviewers will continue using their scoring systems because there has to be a method of rating games (or any consumable product) which is easy to understand at a glance. Finding a reviewer or group of reviewers that you generally agree with and trust and reading their reviews carefully is the best way of finding out if you will like a game. You can ignore the skewed scoring system and give emphasis to the opinions of your trusted reviewers.

Of course, you probably already pre-ordered the game when you saw that gameplay video on the internet so what does it matter what a reviewer thinks? Unless you’re only looking for validation?

Read Part 1 of this series here.

Read Part 2 of this series here.