Over the weekend, I checked out the Freakanomics documentary on Netflix. If you haven’t read the best-selling book or its sequel Super Freakanomics, you really should. It’s a fantastic look at “the hidden side of everything,” where economist Stephen Levitt explains everything from baby name choices to why drug dealers live at home with their mothers, all based on hard numbers.
The film version explores several of the book’s chapters, with one of the more intriguing focusing on Levitt proving corruption within Sumo wrestling. By looking at a large volume of match results, he was able to show that participants were engaging in a sort of you-scratch-my-back-I’ll-scratch-yours system that ensured wrestlers’ livelihoods. And here we thought only WWE matches were fixed.
I decided to apply similar methodology to another area that has long been accused of corruption: video game reviews. Indeed, Metacritic co-founder Marc Doyle revealed last year that he had removed some unnamed publications from the site’s reviews listing because of impropriety. “There’s corruption – people can be bought, absolutely,” he told CVG.
In that vein, I compared Metacritic video game review scores with movie review scores to see what sorts of differences might present themselves. I’ve written before about the discrepencies between the two media as they pertain to the aggregation website – the movie ratings it compiles tend to be from critics employed by mainstream newspapers while almost all game reviews come from dedicated gaming sites.
It’s no secret that these trade publications score their respective products higher than mainstream critics do, which, I’ve argued, is part of the larger problem when it comes to games journalism. As long as trade publications keep pumping up game ratings, publishers have little incentive to deal equitably with the mainstream media, where critics are more likely to be harsher. Dan Hsu, former editor of Electronic Gaming Monthly, explained a lot of the shadiness that goes on at some of these publications a few years ago in a series of blog posts.
So how bad is the discrepancy between movies and games? Very, as the chart below illustrates (click to enlarge). To compile the graphic, I amassed every wide-release movie review between July and this past weekend (35 in total) and compared against every game released for the Xbox 360 over the same time period (51 in total). The results are interesting to say the least.
The average movie score was 53 out of 100 while the average for games was 71. Far more intriguing, however, is the proportion of games that got high marks. More than 60% of games scored upward of the seventieth percentile, while only 19% of movies rated the same. Seven per cent of games scored higher than 90 out of 100, a feat that not even a single film could accomplish. At the other end, only 15% of games scored in the fiftieth percentile or below, compared to two-thirds of movies.
The results suggest several possibilities, the unlikeliest of which is that game publishers are producing very few stinkers and significantly better content relative to Hollywood. Another is that game reviewers are simply more lenient in their reviews – it’s a well-known truism that most games are actually scored on a scale of seven to 10, with anything below a likely kiss of death. Another is that game reviewers are more subject to group think, while film critics are more varied in their interpretations. Still another possibility is that Metacritic is not aggregating proper sources for reviews; movie ratings could be lower, for example, if more trade publications were counted instead of mainstream critics.
Some of the discrepancy can probably also be explained by different rating scales. Most publications score movies on a four- or five-point scale while games are typically rated out of 10 or 100. Even if some of this is factored in, the difference between movie and game ratings is still very big.
The discrepancy doesn’t prove corruption – a much larger Freakanomics-like sampling would be needed to even come close to such a conclusion – but it does suggest problems with how games are being reviewed or, more specifically, how they’re being rated. The law of averages states that the majority of any sort of product is going to be middling, with relatively few exceptions either excelling or failing. Movies definitely follow that pattern with 60% falling between the fortieth and sixtieth percentiles. With games, only about 10% count in that middle area. That just doesn’t seem right, does it?