Over the past couple of months, I’ve noticed a change in the landscape of reviews around the internet. Traditionally, there were a number of large sites that could be trusted to “professionally” review a game. You know who I mean without me needing to name them. I think this has begun to change.
It’s going to be very difficult to talk about this subject without naming names but in the interests of professionalism I will try to avoid specifically calling anyone out on what I believe to be their shoddy work.
It first struck me a month or so ago when I read a review on the website of a magazine which I have genuine respect for. The review was for a game I had just reviewed myself and it was abundantly clear that the reviewer working for this website had not played the game in question for longer than an hour or two.
He mentioned things in the review which simply weren’t true and the only reasonable explanation was that he hadn’t got to the part of the game which made them untrue yet. I was shocked and more than a little disappointed. The website in question even issued a statement claiming that their guy had finished the game and standing behind the review. I understand that it’s important to back your staff but surely not at the expense of honesty or reputation when the evidence is so clearly contrary?
This brings us to the seemingly growing trend of not actually playing the games that you are supposedly reviewed. No outlet will ever admit to this (except us) because if you come out and say “yes, we reviewed that game without actually playing it to its fullest” then you are admitting that you sold the game – and your readers – short. I am reasonably sure of at least one growing outlet that definitely does not play the games it posts reviews for. How can you adequately pass objective judgement on something if you don’t know all it has to offer? You can’t. It’s dishonest and it harms our industry.
In my opinion, this happened just this week with an extremely poorly-written appraisal of an upcoming Move title from one of the big outlets. The reviewer misquoted controls (which have since been edited out, mostly – without referencing the original mistake) and claimed that they were detracting from his enjoyment of the game without actually knowing what those buttons were supposed to be doing. It’s like me saying that a piano is out of tune because I can’t play Rachmaninoff’s third on it. I can’t play the piano and I’ve never tried to learn so why blame the piano makers?
So why bother? The answer is simple: traffic. If you don’t cover as many games as possible you limit your chances at search engine hits and you seem less professional to your readership because you have gaps in your coverage. You see, contrary to what some might want you to believe, high traffic is essential to profitability with a website. If you don’t have the traffic then advertisers don’t want to pay you for adverts, no matter what your advertising model is.
This is the reason for those sites which post multiple sensationalist headlines out of one interview. It’s the reason for most of those sites which post the “announcement of an upcoming announcement” news stories. It’s the reason why many outlets focus on negative stories: negativity is popular. It’s also the reason, sadly, that some sites or writers rig their reviews. All of which brings us nicely to the sites that post artificially low scores for highly anticipated titles.
Now, I’m not suggesting that simply because a title is highly anticipated, it should automatically get a positive score. That is just as bad as the negative scoring trick used by some, it deceives your readers. We try to use the full scope of our review scale on TheSixthAxis and we occasionally moan when it’s clear that others are only using the top third. It is fairly obvious though, that sometimes writers and outlets will purposefully approach a game negatively in the anticipation that it will create a bit of conflict and drive a lot of traffic.
Various outlets have been accused of this and, while it’s not always a fair accusation, I think that it does occasionally have a solid basis in truth. It happens too often and scores from individuals are often too disparate in comparison to the general consensus to be a coincidence. This would be easily accounted for if it was a rare occurrence but certainly not at the regularity with which it happens.
I’ve always been of the opinion that some of the most talented writing on the internet is on small, independent blogs. It genuinely annoys me each year when the nominations for the Games Media Awards are announced in the UK and they’re the same limited selection of outlets, owned by the same couple of publishers or media outlets. Independent outlets and the talent they employ don’t get noticed by most.
So how does that leave the future of games reviews? Well, there are talented people out there who are willing to work for free when they’re not on freelance assignment (Jen Allen and others at Resolution, Lewis Denby and of course, certain TSA staffers all immediately spring to mind) so there’s no shortage of youth and enthusiasm. We just need the established industry giants to take notice. Or we’ll overthrow them.
Note: for the sake of clarity, Lewis Denby has pointed out that he no longer works for free – he earns a living writing about games and doesn’t have the time to invest in unpaid work. He retains that independent spirit though, surely a good sign for the future!
Jimster71
It’s really down to the editor, at the end of the day. The person who sets the standards and procedures that the staff adhere to. It’s a shame that things seem to be sliding downhill in a battle for page hits. The people who want honesty in this regard will soon determine who they can trust and which reviews are worth their salt.
Ryan1991
What makes it look even worse was that the site who reviewed the upcoming Move title badly also gave a world exclusive review for a Kinect game pretty well.
Voganlight
For a casual title…
Kevling
Excellent article CB. I give it 12%. But I’m only using the bottom 13% of the scale. You’d have got the full 13 but you didn’t mention
–
Check back tomorrow when we’ll exclusively be posting the end of this sentence.
arqcan
I’ve been keeping an eye on The Fight for awhile but after having read IGN’s review it totally put me off. When I read TSA review i was back in again. its nice to see that someone out there had the decency to try the full game and not complain and moan when its to hard.
DJ-Katy
I’m not afraid to speak my mind so:
“In my opinion, this happened just this week with an extremely poorly-written appraisal of an upcoming Move title from one of the big outlets.”
This may or may not be the same game and publication we’re thinking of, but I read IGN’s review of Time Crisis Razing Storm two days ago and it was absolutely shocking. He complained about lack of control (it’s an on-rails shooter) and that it’s too short (it’s an arcade port) and had no replay value (the point of arcade ports is to beat your high score). He made various other complaints and ultimately gave the game a 4, when this game includes Time Crisis 4 and Piratestorm or whatever it’s called for free. IGN gave Time Crisis 4 standalone PS3 release an 8.
IGN also managed to give Medal of Honor a 6, citing all the same complaints that most of us could level at Modern Warfare 2, which they gave a 9.5.
My respect level for them has gone right down the last few months (there are various other questionable reviews on there lately too).
nofi
In an entirely related issue, I (on a personal level) thought the IGN ‘review’ of The Fight was utter cack. And wrong.
DJ-Katy
I’m going to go and read that right now, I love a good shitty IGN review :)
DJ-Katy
Hm. I haven’t played the game but tbh the IGN review kind of summarised my expectations. That doesn’t really mean anything of course, but it is more or less in line with the other publisher reviews quoted on Metacritic. But, that’s just a perfect example of why you have to play a game and not trust reviews.
nemesisND1derboy
In all fairness Katy, the IGN reviews of Medal of Honor and Modern Warfare 2 were by two completely different people.
The reviewer of Medal of Honor, Arthur Gies, is one of the co-hosts of the Rebel FM podcast and he stated that he would not have given MW2 the score it got.
Things like this need to be taken in to account when you have a go at a reviewer. I don’t agree with the MW2 review either, but again it was a different person.
In reference to the Fight, I can’t comment on that as I haven’t seen that author on their site before. A lot of it comes down to the reviewers preference.
The reviewer who reviewed Time Crisis Razing Storm, Ryan Clements, is actually a JRPG reviewer. He plays mainly JRPG’s and his opinion of an on-rails arcade shooter would be different to someone else. And again Katy, it was a different reviewer who reviewed Time Crisis 4.
I still trust IGN’s reviews, as they have been spot on with a lot of games, however I also trust TSA.
If there is any site I can’t trust anymore, its Eurogamer. A smattering of 4’s on games that didn’t deserve that score.
DJ-Katy
I’m going to have to disagree strongly here but I’ll explain why.
First of all, a site needs to have a uniform way of scoring games across all reviewers. You don’t give Time Crisis 4 an 8/10 and then a pack of 3 games including Time Crisis 4 for the same price a 4/10, regardless of whether two different people reviewed it. That’s just bad editorial management.
Secondly, one of the editor’s jobs is to delegate reviews to people who are capable of reviewing the games. If Peter asked me to review NHL 11, I’d say sorry, I don’t like hockey, I’m not the right person for the job. If you don’t like the genre you’re playing, you’re obviously not going to enjoy it.
Thirdly, reviewers should be objective enough to be able to play a game and recognise whether it has a user base who will enjoy it, and see the artwork and merits in it, regardless of what they thought. I enjoyed Naughty Bear but I wouldn’t give it a good score because it is riddled with problems that would annoy most people. On the other hand, I thought Joe Danger was bloody atrocious but I can see the fun to be had by the right crowd. The reviewers at IGN totally fail to grasp this concept.
Finally I have to take issue with your mention of Ryan Clements. He is possibly the worst reviewer on the planet, he was not the right person to review TCRZ and he is not a very good JRPG reviewer either. His White Knight Chronicles review – where he complained that it was turn-based among other things – was so laughable that it inspired me to write a proper one for TSA. You don’t complain that an on-rails shooter doesn’t have freedom of movement. If that’s his opinion, he shouldn’t be given clearly known on-rails shooter games to review. Although, Mr. Clements doesn’t seem to actually play anything for more than 10 minutes before reviewing it anyway. I read his reviews these days just to see how woefully inaccurate he has been this time.
nemesisND1derboy
Sorry Katy, I thnki we shall have to agree to differ on this one.
And I find your comment about Ryan Clements not playing games for more than 10 minutes completely unfounded to be honest. The fact that he delayed his Final Fantasy XIII review for longer than other sites is testament to the fact that he plays the games and finishes them. Listen to Podcast Beyond, where he talks about the games he has reviewed and is knowledgeable on the subject.
Then again everyone has their opinion and I will respect yours, I just can’t agree with it.
nemesisND1derboy
And WKC’s reviews range from 8.9 to 4 out of 10. And many reviews have expressed the same problems. The problems he expressed with the battle system weren’t that it was turn-based, rather that the wait time of the “Command Circle” are too long and that, because of the system, there is no incentive in the game for you to change things up. Also the game has no concept of distance, and he supports this by stating: “I have actually fled from a battle, entered a tunnel, turned a corner and emerged from the other side, only to be struck by an attack from an enemy on the other side of the tunnel.” That seems pretty flawed to me.
Tuffcub
TSA: Home to honest, unbiased opions….
..and the occasional recipe.
3shirts
Awesome, I love occasional cake :)
slates16
Agree 100% with this blog, I write reviews myself and I always finish the game even if it puts the rewiew a week behind the “big sites”
ToastEggsBacon
Blame Metacritic. Or, more specifically, the rat race to be “first” on Metacritic in a desperate attempt to garner hits.
I mentioned this in a comment to another topic the other day, but I write for a small, independant website which isn’t Meta-listed through choice. We don’t get enourmous hits or thousands of commenters but we do retain the freedom to say what we want about a game, along with the freedom to take our time with the review and write it when it’s ready, as appose to rushing through the game in an attempt to complete and review it by a specified date for the sake of being amongst the first to be listed on Metacritic.
Like I also said before, I’d be VERY intrigued to know just how many smaller websites listed on Metacritic actually completed games like Final Fantasy XIII or Fallout: New Vegas – huge games that require hours upon hours of investment to complete – before writing their “review”. Our website published a review of Final Fantasy XIII just a couple of weeks ago, with the reviewer in question having invested more than 90 hours of their time into the game.
It all comes down to the individual user at the end of the day to find the sites the trust. I don’t necessarily trust Metacritic scores because there are just too many websites out there who are too quick to post content of questionable quality for the sake of the hits that being listed on Metacritic brings.
Lewis Denby
There are a couple of things I think need pointing out here.
On the subject of reviewers not finishing the game in question, especially in the case of freelancers, a lot of it is down to what’s financially sensible for the writer. It’s a tricky one, and I suspect there are no right answers. But I’ll happily admit, if someone sends me a 50+ hour game to review, there’s no way in hell I’m going to finish the thing before reviewing it.
As a freelancer, you get paid by the word or – more commonly – simply a flat fee. Professionally, these flat fees tend to range from anywhere between about £40 to £200, depending on the size of the website, the number of pages filled in a magazine, or whatever. At the high end of that, then sure, you might expect a reviewer to play the entirety of a lengthy game before reviewing it. At the low end, however, it would simply be unfeasible to do so, because you’d find yourself working at well below minimum wage, and struggling to put food on the table that week.
(There’s a whole other argument, which I think I’d also agree with, which says a player isn’t going to bother chugging along with a 50 hour game if the first ten hours are dreadful. Even if it gets good eventually, can you really recommend it? This becomes a slightly less valid argument if you reverse it, and have a game that’s exceptional for ten hours, then shoddy for the remainder.)
On traffic, I worry you’re simplifying it a little bit. Turning big numbers is of course part of it, but there are other factors in play. Like how much of that traffic is transient, and how much of it sticks around? Transient hits don’t do much good for advertisers, are they’re fleeting visitors who probably aren’t so bothered about what you have to say about the game, whereas dedicated readers will place more trust in your publication. And how targeted is your traffic? Again, advertisers want to be able to market to a specific audience, so having any old hits generated by social bookmarking or whatever isn’t always much use to them.
I also take issue with the idea that an outlying review score is a sign of something fishy going on. We are all allowed to have our own opinions on a subjective craft.
colossalblue
Of course, a number of points are simplified here for the sake of expediency and it’s a general look at how the bond of trust appears, I feel, to be moving away from the larger sites (and magazines to a lesser extent) towards independent – and independently-spirited – sites.
I would agree with the point that 50-hour epic RPGs and the like don’t require completion. Neither do sports games or MMOs which don’t have a traditional completion aspect to them. For Narrative-driven games though, I think it’s very important to see the end of that narrative. For everything else it’s essential to at least explore all of the mechanics that unfold and unlock.
Finally, the outlying review score issue is nothing to be concerned about at all if it happens rarely. It’s the consistency of the occurrence together with the inconsistency in the text that rings alarm bells.
Lewis Denby
“The outlying review score issue is nothing to be concerned about at all if it happens rarely.”
Why should games criticism be any different from film or music criticism, where wildly contrasting reviews from different critics are commonplace?
colossalblue
No reason at all. What I meant was (using movies as an analogy), if one reviewer scored Citizen Kane, Raging Bull, The Godfather and Empire Strikes Back all as 3/10 while his contemporaries were all awarding those films 9/10-10/10 (as is the now-accepted norm) then there is cause to question, at least. On face value, that low-scoring reviewer (although it works the other way too) is either working to a different agenda (which is fine if it’s explicit – like a guy reviewing movies for under-10s) or they don’t have the first clue what they’re doing (also fine, as long as they don’t claim expertise).
So it’s fine (IMO) for a reviewer to not enjoy The Godfather. That can be attributed to personal taste. But if that same guy under-scores a whole string of top-quality movies (as judged by fellow professionals) then surely that’s a little worrying?
djhsecondnature
Well one reason is that it’s just not the same. Regardless of whether you like a style of game or not, there’s no denying it’s graphical quality or how solid the gameplay is. It’s far less subjective than films or movies.
Zephyre
Good article. It’s got me thinking of the last time a game review actually influenced me to buy (or not) a game, and I’m coming up blank.
Pretty much every game I’ve brought in the past year or so has been because I’ve either enjoyed previous ones in the series (TW11, BFBC2) or the people I play with have raved about a game and I’ve picked it up on their recommendation (Saboteur).
Hope you don’t mind me asking, but does TSA review games from retail copies, or are you sent review code by the publishers?