It’s easy to understand why small indie games occasionally turn out bad. Whether the concept is poorly explained or the game suffers from poor design, it’s often because the developer doesn’t have the money or time to do adequate user research.
It is all the more amazing that bigger games, with big publishers behind them, sometimes end up being bad despite having great resources. Shouldn’t they – with all their thorough testing and continuous feedback collection – have been able to predict that?
Earlier this year, Gamereactor spoke to Jonathan Bonillasm, who works for Nordisk Games and has almost ten years of experience with user research. According to Bonillas, the simplest explanation for failed user surveys is that the people responsible for the game sometimes refuse to read the writing on the wall:
“I run into egotistical creative directors who won’t believe the data. They’re totally against it and won’t change anything. You try to build trust, you try to turn it around, but no matter what you do, they won’t believe on you or have anything to do with user surveys.
This is an advertisement:
More often than not, it’s not the fault of a single person that a game flops despite many tests, but it has to do with management issues, different creative visions, or time and money constraints. Factors that can be difficult to handle in large projects. And of course, user research is not just a magic formula that makes all games better. The responses – whether they are qualitative or quantitative – still need to be sorted, and the relevant changes need to be implemented by the developer.
Jonathan Bonillas works for Nordisk Games, which, among other things, is behind Supermassive Games.
Risk of leaks
For the team that carries out the user surveys themselves, there are also many elements to take into account, explains Bonillas. First of all, the testers must be well informed, because it helps little if they give feedback about the difficulty level if the developer is looking to improve the user interface. Another important element is building trust between the user research team and the developer, if the feedback and testing is not done in-house. Finally, it is also important to start early, while the changes can still be implemented relatively easily.
However, this opens up a new problem. During early playtesting, details about the game risk being leaked, which can lead to problems later when it comes to creating hype or managing expectations.
This is an advertisement:
“There are a lot of developers I’ve worked with who don’t want to do user research because they’re afraid [for lekkasjer]. Honestly, it’s a fear you really have to get over. You can ask yourself: “What are we most afraid of?”. Are we afraid that the game will leak, or are we afraid that the game will be bad?”says Bonillas.
That said, Bonillas calls leaks “an ongoing concern”, and explains that measures are often taken in the form of strict NDAs (Non Disclosure Agreements), that phones are put in cupboards during test sessions and so on. But as he explains, few players test to cause problems, and in his nine-year career he has only experienced two cases of leaks from user surveys.
Baldur’s Gate III is perhaps the most famous game to be released in Early Access.
Early Access and creative control
Over the past five years, a new way of testing games has emerged that seems to bypass both traditional user surveys and the risk of leaks. Early Access, which allows players to purchase an unfinished version of a game, can help fund production and provide valuable feedback. But it’s not exactly a bulletproof method.
“I think Early Access is a good method if you don’t have money for user research”, Bonillas says, laughing. “But the difficult thing about Early Access, and the only real downside, is that the feedback can come from anyonekanter. And they can be very confusing. Also, someone has to sort through all that data. It takes a lot of time and effort.”
Early Access also points to some of the more general concerns associated with user research in general. Could the developers risk losing creative control over their game if they constantly receive feedback? And will they have to play with the public?
Jonathan Bonillas explains that this is rarely the case, as user research is often aimed at improving specific elements of the game, compared to more general market research that tries to identify what is popular or trendy at the moment. But as he reveals with an anecdote, developers sometimes take the feedback too much to heart. Once there was a small studio that, after being presented with user research, simply dropped the existing concept and developed a completely new game!
When feedback from fans and playtesters is ignored, the product rarely turns out well.
Saved by marketing
As a final question, we asked Jonathan Bonillas if he had ever worked on a project where all the feedback from the users was negative, but where the game was still well received by the players. Perhaps unsurprisingly, he answered no, but he did reveal that sometimes a game can still sell well even if the feedback from the testers has been predominantly negative.
“There are games I’ve worked on where we knew it was going to be bad, but the marketing is so good that it still sells five million units. Sometimes the company sees that as a relief. “Great, we sold it, we got the money back and made some money. We knew it was going to be rubbish, but at least we got our money back. Sometimes it is, especially with the big budget titles. And it sucks, because it doesn’t motivate me”, he explains, laughing.
2023-09-03 11:44:39
#benefits #dangers #conducting #user #research #games
