The Love/Hate Relationship Developers Have With Game Review Scores

Few things are certain in life other than death, taxes, and people getting upset about video game reviews. While anger generally comes from fans that are upset that their pre-order wasn’t justified by a high score, occasionally a video game developer will openly criticize a review. While these cases are few and far between, it’s clear that developers have a very strange relationship with reviews.

After all, developers are the ones that give outlets review copies, and then hope that positive reviews will then boost the sales of their game. Good reviews can push an interested gamer over the edge, while negative ones can scare away would-be buyers. A lot is potentially riding on reviews, especially for games that don’t have huge marketing budgets attached to them.

So, what do developers think about the current climate of reviews, and how do they handle criticism?

Value of Reviews

There’s no denying that gaming coverage is evolving as can be seen with the rise of Twitch and YouTube. This has led to many discussions over whether or not gaming reviews are still an important tool for consumers looking to purchase quality games. I raised this question to several developers, and asked them if reviews are still useful in today’s gaming climate.

Assault Android Cactus designer Sanatana Mishra recognizes that the climate is changing, but still finds reviews to be useful. “As a creator I think there’s immense value in having your work critiqued and getting to look at it from the outside in,” said Mishra. “It helps me understand things about my game that would otherwise be impossible because of how close I am to it.” He also believes that reviews are “immensely important” for indies in particular. “We have no way of reaching people except by impressing those who’ll take the time to look at our game and tell others about their (hopefully) great experience.”

Other developers feel like reviews have value, but they may have to be tweaked in order to maintain relevance over time. “I’m a huge advocate of critical writing about video games and for players being informed about the quality of new releases at launch,” said No Goblin founder Dan Teasdale. “But it seems like the standard ‘review with a score at the end’ is the worst version of both of these things in 2017.” Teasdale says he prefers to read launch day impressions over reviews as they are “less likely to fall into the usual review template” and “reflect the writer’s actual impressions rather than an overproduced dissertation that has to hit a bunch of bases.” As far as traditional reviews go, he says that “Kotaku’s scoreless model of placing an info summary box is a good middle ground.”

Review Aggregates

Another wrench thrown into the mix is review aggregates. Websites like Metacritic and OpenCritic compile dozens of scores (all written under different grading criteria) and turn thousands of words of criticism into a single number. It’s incredibly useful for a broad look at a game, but all the nuance is lost. This number can be stressful for developers for a number of reasons, as many games with a low score are automatically dismissed by gamers and sometimes financial bonuses can be tied to getting a high Metacritic score.

“I think they were planning on finding an excuse to fuck those New Vegas devs out of that money no matter what happened,” speculates Actual Sunlight creator Will O’Neill, referencing when Obsidian missed a sizable bonus payment by one point on Metacritic. O’Neill has never been one to mince words, and he doesn’t feel like aggregates impact him any. “I’m not making things that I think are reasonable to expect most people to like,” says O’Neill, whose best known title is a word-heavy game that deals with depression and suicide. “Anyone who knows their tastes fall into the minority ignores those sites regardless, and I think they have a greater impact on forum slapfights than they do on actual consumer behavior.”

Other developers do worry about these numbers, and they can actually make a developer disappointed in a positive review. “It really sucks when you get a great review, but it’s below the average,” says Sanatana Mishra. “It turns what should be a lovely moment where [a critic] gives our game 4/5 stars and exclaim it as being great into a weird moment where you realize this positive thing is going to hurt your game’s perceived quality.” Mishra believes that aggregates are “problematic,” but recognizes that aggregation would occur even if there weren’t sites built around it. Personally, he prefers using OpenCritic due to Metacritic’s vagueness in weighting sites, and says that at least with the former, “you know what you’re looking at.”

“I think the concept of review aggregate sites are tarnished by how miserable Metacritic is,” says Dan Teasdale. The mind behind Roundabout would rather see a Rotten Tomatoes-style site, and says that “while it’s not perfect, I think OpenCritic is doing a far better job of the ‘combine to a single score’ job than Metacritic is.” Teasdale also finds it frustrating that Metacritic allows “Wordpress blogs nobody has ever heard of” to impact scores, and says that the site has failed to adapt with the time. Due to Metacritic not allowing updated scores or outlets to adapt their score to the overall Metascore “it just ends up creating a score that’s not representative of the general mood amongst reviewers, which defeats the point of even aggregating.”

Launch Day

“Finishing and releasing a game is one of the most stressful things you could ever do,” reveals Sanatana Mishra. “Before even thinking about embargo day you’re probably already in an awful state of mind, and you will only get more nervous as that moment where you find out the reception approaches.” Despite the nerves, Mishra says he ended up reading “absolutely everything” about Assault Android Cactus when it released. He says that while he reads user reviews, tweets and forum threads, he’ll often read full reviews multiple times throughout a day to make sure he understands them. “One thing you get very good at as a developer is understanding the difference between emotional responses and logical reasoning because people are always right about how they feel, but not so much about why they feel that way. It’s awfully tempting to look at a negative review and dissect their reasoning to ‘prove’ they are wrong, but ultimately you have to accept if someone doesn’t like your game it doesn’t matter if they can’t articulate why.”

Similarly, Dan Teasdale also fights nerves while waiting for reviews to hit on launch day. “I’m absolutely nervous waiting for the first scores, usually to the point where I can’t really focus on reading everything slowly and slam through the text trying to get a sense of what people liked and didn’t like.” Teasdale also says that since he sets embargoes the day of launch (his reasoning being that “if you read a good review about our game, you’re able to go and buy it straight away”), he can’t really dwell on reviews since launch days are often filled with “good distractions” in the form of emergency patches and responding to the community online. “At the end of the day though, I’ve made peace and stand behind whatever we’re shipping – so at best reviews are a minor ego boost, at worst some time to crack open a beer and bookmark them for when we do the project postmortem.”

How Useful Is Feedback?

“I’ve been unhappy with every game I’ve shipped,” reveals Dan Teasdale, when asked if feedback from reviews are valuable. “Even the ones with 90+ review scores.  I think that’s pretty common among game developers — all we can see are the problems and cut features. If a reviewer is complaining about something, it’s almost definitely something we’ve worried about too.” While Teasdale feels like most of the issues are known by release, he says that small things tend to be overlooked and then caught in reviews. “Restarting mid-mission in Roundabout is a great example of that, we read that complaint in the Gamespot review at launch, and added it next day in a Steam patch.”

Sanatana Mishra says that a “good critique of your work is truly an invaluable thing,” as the external perspective allows creators to grow. “It’s true that most reviews will just cover the broad strokes of your game and not go beyond what you are surely aware of but I think that’s often what a broad audience wants, finding a balance between giving the crucial information a player needs to understand a game and going in-depth with the nuance of that game must be a very tricky thing for a critic.”

On the other hand, Will O’Neill says he doesn’t “expect to be affected much by anything that reviews have to say” regarding his upcoming title Little Red Lie. While he says that reviews can be helpful for spotting aspects of his game that were misinterpreted, “it’s difficult to know if that means it is a problem on a larger scale.” O’Neill also doesn’t mind being misunderstood by the “majority of people” as long as his game has “the right kind of impact on the people who it will truly speak to.”

O’Neill finished the interview by saying that he hopes gaming journalists and critics “are doing what they’re doing to earn a living.” He believes that “if they’re doing it with the intent of shaping and influencing what game developers are making, I think they’re going to end up very disappointed. It’s an excruciating amount of work, and I’m not doing it for anyone or anything but myself.”

Essential Reading:

• Buying Games Day One Is No Longer the Tempting Proposition It Once Was
• Top 5 Reasons Why PlayStation Gamers Might Want a Nintendo Switch
• 10 Games That Would Make Good Movies (No, Seriously)