by Martin Wakeley
Game evaluation is a sensitive subject. In theory, it's part of a mechanism to ensure the game is in an optimum state when released. In practice, it often leads to the Development team feeling alienated and disheartened.
My experience of the Game Evaluation process over the last 20 years has ranged from elation, to acquiescence, to depression.
Nintendo, as you would expect, do Game Evaluation extremely well. This is underpinned by a desire to act on the findings and produce the best possible product for their customers. Their drive to release games when they are ready, rather than when a sales window is open is admirable. It promotes quality and value for money for the customer and stimulates ownership and consistency in the Developer.
Nintendo don't base their findings purely on graphics or cut-scenes. They get players who understand games to play and pass observation. One of my earliest encounters with Nintendo was during the making of Blastcorps. I thought games should be difficult because that's what I enjoyed: one day we received some feedback from a tester in Japan saying he had attempted a level over 100 times and it made him want to cry in frustration! That is powerful feedback and we made the changes accordingly.
Some forward-thinking publishers like to market research the concept for their games to test consumer appetite. This is great in principal. Fundamentally there is no point in making a game if no one is interested in playing it or the market is already saturated. The key to these campaigns is to act on their findings. All too often, for financial and political reasons, sound findings simply get ignored.
Once a game reaches a playable state then large-scale playtests are fantastic. There is no better way to analyse how a game plays than by looking at comprehensive data about which levels are popular and which features are used. It removes all the subjectivity from the discussion. Telemetry has revolutionized the way we can evaluate games. Back in the day, we put our game in a box, sent it to the shops and never really heard any more apart from reviews and the occasional angry letter.
QA play an important role in evaluation. They provide instant feedback to new features, and invariably know the game better than anyone. A good QA team supports the development team and feeds into the design process. The big caveat to this is that they are insanely good at playing games - and they know all the exploits. They are not always the best people to represent the 'Average Gamer'.
It's clear there are many productive ways to evaluate games. So how come questionable games are released? Surely someone noticed this during development?
Evaluation really becomes pointless when there is no desire, time or resources to implement the findings. Eleventh hour reports criticising features are useless where there's no time to implement changes. Time and money constraints are the nemesis of quality at the best of times. The irony is that the people requesting the changes are usually the ones controlling the budget.
As a wise man once told me, in Game Development you can have 2 out of the 3 tenets of 'Good', 'Quick' and 'Cheap'. Publishers invariably want 'Good' but because they have chosen 'Quick' and 'Cheap' this can't happen. Utopist game development assumes unlimited time and resources. The reality is often much different and this leads to disenchantment for the development team. No-one wants to hear that quality is being compromised to hit a quarterly production slot.
These days, with the cost of projects spiralling, it's way too risky to release a game without evaluating all aspects at every stage. The key to making this work is to factor it in from the start and be prepared to act on the findings. Nintendo do, and that's why they make great games.
It's a noisy world out there and it's tough to get a new company off the ground. If you would like to see us succeed, you can help spread the word by sharing this article. We can't offer anything in return except our thanks and a promise that we'll always try to make great games.