We first heard rumours and accusations of this last week but without anything solid to base it on, we decided to give the story time to develop. As it has transpired, it was all true but certainly worth taking a more complete look at the story now it has had time to develop.
A user, going by the name of Avanost, posted a glowing user review for Dragon Age II on MetaCritic. Amid the flurry of extremely negative user reviews that were being posted there around the release date, it looked out of place. This prompted an investigation from Reddit user, GatoFiasco who was concerned with the ethics of the situation.
This is a matter of ethics and integrity. A consumer requires objective information in order to make an informed decision about purchasing a product. If the line between editorial article and product review is skewed, then the consumer is being deceived at the cost of their eventual trust and loyalty to the company responsible. This is why disclosure of industry ties is necessary to avoid even the appearance of impropriety.
Meanwhile, a senior Public Relations Representative for EA (BioWare’s parent company) has defended the move, likening it to a politician who votes for themselves on election day.
We’re slightly torn on this issue. On the one hand, GatoFiasco is perfectly correct, it is imperative that a system exists to provide consumers with as objective as possible information regarding a product. Is that best done with a user rating system so easily open to orchestrated abuse in its early stages? Certainly not. That’s not to say that Metacritic’s user rating system is at fault, it’s not, simply that all user ratings systems are open to abuse until there is substantial data to be counted.
It’s clear that there were a large number of Dragon Age fans that were disgruntled with the changes made to the series, the Metacritic user rating was below three well before the game was actually released, presumably based on nothing but feature points from published previews and press releases. Is that an objective way for anyone to rate a game? No but it does effectively demonstrate a group’s desire to show their frustration at quoted changes, perhaps even without playing the game for themselves. Isn’t that an actual abuse of the system?
The question of whether a BioWare employee should be allowed to enter his own opinion into the user rating system is also a strange one. If you’re going to champion an open, collective user rating system then surely it has to be open? Why should we expect anyone to be excluded. Surely that’s the down side to user ratings: they are easily tampered with and manipulated, at least until there is a massive weight of collected data. Should we somehow restrict that system so that it is only allowed to be manipulated by people who aren’t involved in the games industry? Surely would yield less objective results than allowing all opinions to be counted?
It’s a tricky question to navigate and, as ever, we’re keen to hear the thoughts of our own users on the subject.
You can read our completely independent review of Dragon Age II here.
Kovacs
I pay very little attention to the user reviews on Metacritic. I’m not being snobbish or insinuating a user can’t review a game. It’s just that the whole area is a quagmire of people often expressing their discontent toward a game/publisher/developer/even console that it’s impossible to quickly pluck the valid reviews from the noise. Shame really.
nofi
If you could RT on TSA, I’d do it here.
User reviews are so fallable they’re basically completely useless.
zb100
Absolutely spot on Nofi. Ever since I first became aware that there are companies that exist solely to “aggressively market” ie publsih “user reviews” for products, I’ve never given User Reviews a second thought.
I will, however, speak to friends & colleagues and read information from content-providers and users on trusted sites/forums such as… oh! The Sixth Axis!
That said, I usually completely disagree with everything anyone says as I’m fickle/have my own personal tastes. It’s never intended as an attack on somebody else, just an expression of opinion – of which there are many.
xdarkmagician
just user reviews? What about the professional magazines and websites that have parent companies that sell games. The ones that helped create the 7 is bad standard? Surely any review that serves a hidden agenda is just as just as fallable as user reviews that are fanboy-ish in nature. I find that if you take the time to read all the user reviews even the outrageous ones that together they paint a pretty good picture of a games pros and cons. I’d also like to think that all user reviews are from amateur game players and have no affiliation with the game. I know this isn’t the case but in my opinion it should be.
djhsecondnature
Users are far more likely to rate something if it’s bad than good, just to ‘warn’ others. The proportion of middle ground responses are also far lower. Overall it’s very hard to trust user reviews. For me the only exceptions are IMDB and Amazon, although I read the text with the reviews on Amazon as well.
bunimomike
Couldn’t agree more. Flawed from the off. People are quick to complain and rarely take time out to flatter.
xdarkmagician
While I agree user reviews are more negative, they are a good balance to professional reviews that tend to only focus on the positive. I think the goal of each is to counter the other.
bunimomike
What a pathetic fuss for them to kick up. Have they allowed for the fact that a staff member might think the game’s a bit crap and go say so on the User Reviews? If he loves the game, good on him! Go write up a noteworthy opinion wherever he fancies.
How can user reviews be anything more than pathetic (on the likes of Metacritic) when people are reviewing the game who haven’t even bought it!).
Aquastyle
Quite the foolish move there by Aganost if you ask me. His review can’t have had too much of an impact on the score, which I’d imagine very few people base their purchase on anyway. And he forgot the most basic of rules; don’t get caught.
Certainly all PR can’t be good?
DrNate86
Bit embarrassing, but as you said, user reviews are such a quagmire of biased opinions, one more is hardly going to break the system. If it had been a published review then this would be a different matter, but that’s like comparing a news report to a man standing on a box shouting at people walking past.
Person678
I can’t blame him. If I was a BioWare employee and saw the user score trolled to well below 3 long before any of them had even played the game, I’d be incredibly frustrated.
DJ-Katy
Can I just point out that you can’t give a user score to a game on Metacritic until the day it is released. Until then, it shows a countdown timer.
Any scores given to a game before release have to occur in the midnight-9am window assuming the shops are all shut til 9am.
DA2 had a 3.6 score based on 70 reviews at 9am on the day of release. It’s 3.8 based on 1000 ratings was some days later. Nobody cares about 70 ratings, but 1000 is a substantial sample. The average review and average user score distance is far greater on DA2 than it is even on MW2 or Blops, where you expect fanboy downratings.
Charmed_Fanatic
Thanks, thought it was weird people reviewing a game before they even played it
nemesisND1derboy
But again, you do not know that these people have played the game. Just because fanboys protesting against what I think is an excellent game, personally, posted at 9am on the day of release doesn’t mean that’s the only time they posted. You have to take these user reviews with a factory of salt. You simply do not know and have no way of knowing 9f these people have finished the game or even played it in the first place. I could easily pop over to metacritic now and post a perfect 10 review if I wanted. I’m not going to, but it’s that easy that I could.
nemesisND1derboy
And I’m assuming Peter meant the UK/European release. It was released in the U.S. on Tuesday, 3 days before the European release, which I assume is what he meant.
DJ-Katy
How do you know the professional reviewers have played the game?
Which of the two groups are more likely to be influenced by EA carrot-and-stick implicit corruption?
What is the difference between someone who writes a user review, and someone who writes a review as editorial content for a gaming site? In the case of the independent sites, very little.
I’m not suggesting metacritic user scores are an accurate barometer, but the whole issue of user scores being valid or invalid, vs professional scores being valid or invalid, is a lot less clear cut than most of you seem to think.
nemesisND1derboy
Oh come on Katy, you mean to tell me that TheSixthAxis reviewed the game and didn’t even play it? It is a hell of a lot more clear cut than you think. You can be guaranteed that most if not all the sites actually accepted for metacritic have played the game, the same can’t be said about the users. I don’t know what your problem is here, if you don’t agree that the game is worth 9 out of 10, that’s your opinion. But your opinion DOES NOT reflect the opinion of everyone else. I love the game, Watchful seemed to really enjoy it too, as it seems have a lot of journalists. If you don’t think that Watchful or another reviewer did in fact play the game, then e-mail Peter and make a complaint. There is no point insinuating that any reviewer didn’t play the game without any evidence to corroborate this. I believe it’s a very good game and it’s worth 9 out of 10. In m opinion, you are as wrong as anyone can be on this subject. Spreading conjecture is not the way to get your point accross. If you don’t like the game, well for you. But I can guarantee you that 95% of the user reviews are from fanboys and from people who either don’t like the genre, or picked it up on false pretences.
DJ-Katy
I haven’t played the game and I haven’t passed any personal opinion on it other than that I didn’t enjoy the demo, which is irrelevant to this discussion. I have only pointed out the disparity of editorial and user review scores. Whether the game is good or not is a moot point, this is about whether the views of users are any more or less relevant or valid than those of the gaming press. It has nothing to do with Dragon Age 2 per se, you seem to have missed that point.
To the particular game, I have no doubt Greg played it, I’d expect nothing less from him, I didn’t say anything about TSA specifically in this thread either as you seem to imply, I said that in many cases the reviewers writing at independent gaming sites are in essence no more qualified to write reviews than anyone who posts on metacritic. TSA is an independent gaming site, but it’s only one of many.
As to your assertion that sites chosen for Metacritic play the games, that theory has been disproven several times by reviewers for major sites listed on Metacritic being caught reviewing games when their achievement or trophy collections reflect that they haven’t gotten particularly far in them. Those individuals continue to review games for those sites.
You are naive if you believe that publisher relationships don’t have any impact on reviews on larger sites. Publishers are the hand that feeds: they provide the review code needed to get the reviews out in a timely fashion, on the day of release. Upset the publishers and you instantly become uncompetitive versus the sites who are shrewd enough to stay in bed with them. This is no different from any other industry: witness Top Gear not being able to get a car from Bentley this season because they were expected to slam it. In the gaming industry, Eidos were old-school masters of pulling a tantrum when a game got a bad review. More recently, the publisher of Two Worlds II demanded reviews be removed from sites and tried to suppress publication of others when they discovered the game was receiving lower than 8 out of 10 review scores in some places.
The gaming press is highly political. My trust in reviews at TSA was based largely on the fact that it wasn’t political here. Nowadays it is, and whether intentionally, by co-ercion, or I hope, most likely accidentally due to the fact Greg was in the sizeable minority who did enjoy DA2, sucking EA’s cock a bit will never hurt, will it?
DJ-Katy
Incidentally the reason I remember f.ex. the Eidos thing is because I started writing reviews for a gaming magazine in 1995, it was corrupt then and it’s corrupt now, and most people here including some of the staff will have had way less time to digest how gaming journalism works in the real world. Hell some of them were still learning to read and write.
yogdog
Phil, you can’t guarantee that 95% of people who reviewed on Metacritic were fanboys etc, stop talking out of your arse. The way you’re speaking, the only persons opinion that matters are the reviewers, and not the people who read the reviews and actually buy the game. In my opinion, the user opinions are more important than any reviewers opinion as they are the ones that have paid money for the game and want to get enjoyment out of it. The gulf between user and reviewer scores here is very worrying, and it seems to me that ‘professional reviewers’ scores on games are out of touch with public opinion.
nemesisND1derboy
You have basically contradicted everything you have said on the matter in this post. This has everything to do with Dragon Age 2, since you said that as TSA reviewed it highly that they have just tried to go with every other reviewer. In fact, the point you made about the disparity between user reviews and journalist reviews was basically centered around Dragon Age. It is such a redundant point as these are opinions that we are talking about. If you haven’t played the game, how do you know that it isn’t a 9 out of 10 game. And you very clearly don’t agree with that despite you saying you have “passed no personal opinions”. If you didn’t have any personal opinions on the game you wouldn’t have posted that attack on the review, which it definitely was. Everyone point of argument you have raised in this discussion you have immediately contradicted in your proceeding post. The disparity between user reviews and professional reviews has in fact nothing to do with the Dragon Age 2, which you have just stated in your last post, which begs the question; why even bring it up at all and have a go at Greg and his review? And your comments about the authenticity and integrity of journalists’ reviews have again got no relevance in relation to Dragon Age 2. And as for Greg being one of the minority of people who enjoyed the game, you again have no idea who liked the fucking game. The reviews on metacritic do in no way represent the entire gaming population. Just because the majority of metacritic users who “reviewed” the game “didn’t like it”, does in no way mean the the majority of people who have played the game didn’t like it. Your belief that people didn’t like the game has in fact no factual basis whatsoever. Your whole argument is just ridiculous and highly contradictory at this point. I will state again what I stated previously, I am really enjoying the game and think it’s great. Despite what you say in your previous comment, saying you don’t have an opinion on the game is utter bullshit and I think you actually need to play more than the demo, which, like the Mass Effect 2 demo, doesn’t represent the quality of the finished product in the slightest, before you pass judgment in the game. It is but a miniscule portion of a huge game. I would not be surprised to hear that most of the metacritic posts are from people who have only played the demo or have not gotten far enough in the game to experience what the entire game has to offer in terms of quality. And publisher incentives are a total non issue here. Again, if you have evidence to suggest that a reviewer has in fact taken bribes to alter the score of a game, then I suggest you write to the site in question and voice your complaint instead of moaning about it in the comments section of this site.
nemesisND1derboy
Jesus Christ John, telling ME to stop talking out of my arse? I seriously have to question your motives in relation to this discussion, and you know why. If you would actually read and think about what she is saying, you would see who the fuck is actually talking out of their arse on this matter. Like I said, if you question the integrity of a review, e-mail the reviewer of the site.
yogdog
Phil, I’m with you on that the demo does not reflect the game in many occurrences, such as Dead Space, or more recently, as you have said, Mass Effect 2. However, I’ve played DA2 for a couple of hours at a friends and is nothing on the first and deserves the low scores it is getting in MY opinion. Questioning my views and motives here is unfair and undeserved, MY views are MY own. Katy has raised valid points here yet you have just shafted them as you believe that 95% of the users (people) who have played the game and posted a review on metacritic are ‘fanboys’ and so on. Yeah, some of them might be, but that’s a hell of a lot within such a short time and some of them have to be honest people who honestly disagree with the road the game has taken, and that it has scored too highly. The gulf between user and reviewers here is absolutely huge and has led me to question the integrity of some reviewers. From my view here, because you think the game is great and you agree with the review, you can discard what users think and question my motives. You could be in the right here, but I don’t think you are.
The Lone Steven
@Katy, TSA are honest in their reviews. I’m pretty sure that CB would be suspious if every review was a 10/10 and he probably pull those reviews and question every staff memeber. Why would TSA accept a bribe from EA? TSA wouln’t asit is a site for gamers by gamers with a very strong community. Each reviewer is required to play the game long enough to gain a proffessional opinion. If they did try to complete every game they had to review, then we would have reviews that are out a month or two after the intial launch.
Okay, larger site may accept bribes but i willing to say and i bet my PS3 on this that TSA refuses all bribes. A review is basically someone opinion. Katy, you said that you hate the game because you played a demo of it? Isn’t that you are accusing TSA of doing. If you hate the demo, fair enough but please don’t accuse TSA of being corrupt without evidence to back it up. I think CB makes sure that each review backs up the score. have a good day.
DJ-Katy
Omg, what part of “I haven’t played the game, I have no opinion on the game, I don’t hate the game” do you people not understand?
My attack on the review was based on the increasing amount of publisher dick-sucking going on, and as to your ’email a complaint’, I complained about it there and got the expected shafting from TSA _fanbois_, although Greg was kind enough to write a mature response to my rant which I accept.
I have no idea if the game is good or not, and I don’t really care, I only care that we get objective editorial and I really don’t believe we have in this case; according to Greg we got his honest opinion and so that is fine, but I seriously don’t think that is the case across the whole gaming press for this product. Why is it not about Dragon Age 2 per se? Because if reviews of DA2 are misleading, so by consequence are reviews of other games.
To the last poster (unfortunately I can’t see your name while typing a reply), I used to write reviews here. I reviewed a lot of the big RPGs last year for this site, so I know the review policy as it stood at the time.
Erroneus
I laughed at the subtitle.
acridweasel
“On the one hand, Avanost is perfectly correct, it is imperative that a system exists to provide consumers with as objective as possible information regarding a product.”
Correct me if I’m wrong, but was it not GatoFiasco who was questioning the reviewer and the reviewer was Avanost?
colossalblue
indeed, I’ve fixed the post to reflect that mistake. Thanks for spotting ;)
Foxhound_Solid
Pinch of salt time – bet he feels a twat tho…
billsmugs
The only user reviews I trust are those on gamefaqs.com, the metacritic ones are awful (check out the one highlighted here: http://awurl.com/JgUTP934L, or the pages of Black Ops reviews for examples)
DrNate86
I like the Gamefaq user reviews, they all seem like some actual time and thought has gone into them.