Play to the Audience #3 - The Test
How to test a video game
This is the third part of Play to the Audience, a series on how games form and maintain a connection with players. You can start from Part One here.
Introduction
When trying to build games that truly engage with an audience, there is one key method of opening a dialogue during the development process: testing.
Most game developers know you should playtest your work "early and often" but this straightforward advice can lead to a world of confusion. It's not uncommon to hear "we tested throughout development" from devs when they haven't conducted a single focussed playtest and thus are completely in the dark about the appeal of their game.
In other instances, the feedback that developers have received will be centered entirely around UX, onboarding and polish with very little insight into the core game design and whether it meets the needs of their players.
In this piece, I'll focus almost exclusively on playtesting, as there are already a wealth of resources out there for those exploring QA and UX. I'd encourage self-taught devs who don't have much software development experience outside games to look into general best practices and newer methodologies such as test-driven development to see if anything there is helpful to their process. It is imperative that you develop a testing regime which will help you get your work out there in a stable and relatively bug-free state: I've seen many promising projects get torpedoed at the last minute by a buggy release.
As ever, I will have a bias towards PC-centric indie development. In the mobile domain, there is a much greater focus on data-driven decision making once a game is live. There are lessons to learn from this for PC devs - even those with a pay once business model - and I will discuss some of those very briefly, but my main focus will be on fundamental testing rather than the more abstracted and nuanced area of analytics-driven game design. Analytics are just another form of input - the process is the thing that matters.
A quick note before we get started. Testing can often seem like a thankless boring slog: "I know the problems with my game so why do I need other people to tell me? I'll just show them when it's finished". Additionally, some devs are resistant to the idea of "focus testing their art", fearing that it'll dilute their creative intuition.
If you unlock the right forms of testing for your game, as well as asking the right questions, it should be a liberating and empowering experience that turns game design into a more concrete and rewarding process. If you are designing a highly original game in particular, much of your early efforts will involve feeling your way in the dark - testing and discussing with a supportive audience who want the best outcome can provide much-needed illumination. Testing is ultimately a process of accumulating information - your creativity and judgement will be the ultimate arbiters of how that information is deployed.
Testing should take place in broad phases that contain repeated cycles of iteration, deployment and evaluation. You should be clear about your goals for each phase and make a considered decision to progress beyond it - treating these phases as a series of gates that your project must pass through is a useful framing.
Everything that follows is a suggestion. Your process might require a big engaged community from the very first prototype, or you might be a designer who needs carefully controlled feedback from just one other person until surprisingly late in the process: I have seen both work effectively! The real goal here is to keep trying things and learning what works for you: however, I would encourage moving gently out of your comfort zone. Game design is always an experiment, so don’t be afraid to try new things and then analyse the data.
Prototype Testing
Prototyping involves building a barebones playable mechanical subset, then evaluating it. In the indie game sphere, developers will usually test this themselves and with a few trusted friends or colleagues; larger companies may have more formal processes.
Very few people are truly good at testing early prototypes. Many gamers need visual excitement, functional reward systems and other supporting elements in order to make a judgement about a game. That's fine - and such players have a very important role in later testing phases - but unless you're willing to take the time to train people in evaluating a prototype properly, you'll likely need to be selective about who you trust to give you actionable and relevant feedback.
One of the worst pitfalls in game development comes about due to the tension between iterating and communicating the gameplay: trying to demonstrate the value of a game too early to external stakeholders can lead to distraction, skewed perceptions and misaligned priorities. New unbalanced features might be ideal in the long-term but negatively impact the moment-to-moment experience of playing the prototype - this is where you need to be confident in your overall design vision.
Art-centric or inexperienced teams often try to polish their prototype immediately - they want it to seem impressive at first glance. This is almost always a mistake. The objective here is to test the absolute fundamentals of the game and not how those fundamentals are communicated or presented. It's unavoidable that "feel" and some very basic UX concerns come into play, but again you should avoid leaning too much in this direction.
If your game is a platformer, you would test movement mechanics, jumping and attacking enemies. If it's a turn-based strategy game, you would test a mid-game combat turn to figure out if there is the potential for the player to make interesting decisions. In a simulation game, you could test a very small subset of core systems, using mock or randomised data to provide some kind of useful background noise.
Personally, I enjoy testing prototypes with ultra basic "programmer art" or primitive greyboxing - you don't need to intentionally make your prototype look terrible, but avoiding anything that's too distracting at this stage is ideal. Your testers should be ignoring bugs wherever possible to focus on gameplay - quick restart or panic options to help with this can be great.
The goal of this phase is to answer the critical question "what is the game"? We discussed this back in Part 1 - you need to clarify that the basic interactions you are presenting can grab the attention of players but also have the potential to be fun in the long-term.
Alongside this mechanical design, you can start testing your concept. Once you know that your small testing group loves the core gameplay, you can start playing with settings, themes and visuals.
Questions
What are the player's short, medium and long-term objectives?
Can we identify if we're working in an established genre?
If we're doing something radically new, what are the closest touchstones that players will use and how can we leverage them?
Are the basic interactions fun?
Are any of the basic interactions particularly annoying or uncomfortable?
Is planning to meet those objectives enjoyable?
Is the player making defined moment-to-moment choices with clear consequences?
Is the player making real trade-offs?
Are they reaping what they sow by doing so?
If any decisions become trivial, do they also become boring?
Does the game have the right balance of intensity and relaxation?
How does losing feel? Does that need to change or be amplified?
What is the "tone" of the game's most fun elements - are you zooming around at speed or being patient and choosing your shots for example?
How can that tone be accentuated in future iterations?
Is the amount of content that this game needs to shine feasible for our team?
Do we want to work on this thing at all - does the romance of the idea fade as soon as we see a tangible version of it?
A Note on Concepts
It's difficult to test a game's concept and visuals in isolation without having a somewhat polished build that others can try out, or enough content to put up a store page. Classic online marketing tactics such as running parallel ads for different versions of a product or setting up "placeholder" (a euphemism, in this instance, for fake) landing pages are challenging in games, where the audience often expects more substance. There are some publishers who will sling up a load of different Steam pages to test appetite but this has a strong sense of something that Valve in particular may eventually clamp down on! This is an area that - outside mobile - is extremely underdeveloped. It's likely easier for developers with existing audiences to take the temperature of their community on very early ideas but there is a huge potential advantage here for devs who can come up with clever approaches.
Feature Focus
This is a very tricky phase and perhaps one of the most complicated iterative testing loops. Your objective here is to assess the game's overall feature-set as you build it up from the basic prototype: what needs adding, what needs tweaking and what needs cutting? This is about expressing and developing the concept, rather than evaluating whether it will work at all, and as such it builds on the foundation you constructed with the prototype.
For some complex games, such as simulations or deckbuilders, it can be very challenging to figure out exactly how much content or how many systems will be needed, as the final gameplay tends to emerge from combinatorial effects. This is where technical frameworks that enable you to iterate rapidly can really help out - it is worth planning for this from the outset.
You might consider widening the testing group at this point (particularly if you are inexperienced with your game's genre) but as you're still early in the process, a lack of art and UX polish might prove to be a barrier. Also, as with the Prototype phase, it can be hard to find testers who can limit their focus to productive areas.
This is the time when you want to start actively thinking about retention and holding attention. You will likely have at least decided on a theme and visual direction, so testing your features for consonance with those can be very helpful. Continue to trust your own design instincts though - taking the wrong feedback here can throw you off course.
Questions
What will players want to do in the game that they currently can't?
Do you feel as if you are able to play the character you want in the game?
Does any part of the game risk becoming underbaked?
Are we trying to make two (or even many!) separate games at the same time and stick them awkwardly together - should we focus on one thing?
As features are added, do they increase the possibility space in an interesting way or simply add noise?
Is there a specific move or strategy that feels particularly good to use - should we lean into that direction more?
What do we need to make this a "forever" game that players will keep coming back to?
How do we keep fighting the boredom factors we established when prototyping - what makes players want to quit?
What will it take to finalise our art direction?
What could we just cut right now?
Do we have enough ideas for future content?
Are we aligned with our players - do we love making the sort of content they will want?
Gut Check
Once you have a defined feature set from the Feature Focus phase and built out sufficient test content, it's time to start evaluating the game as a whole. This is likely a point of no return for the project, so it is worth giving yourself the time you need.
In order to complete this phase effectively, you need to start asking hard questions about your current build. Is it really ready for people outside of the dev team to experience? Does it all hang together and work in concert, or does it require caveats, assumptions, excuses or fantasies that exist solely within the brains of the dev team to get through a play session?
This form of validation can often just require a mental reset from the team - a brief period away from the game followed by some quick intensive testing can really help. The Gut Check is the gateway to alpha - it's about calibrating your team's own perception of the game before you're ready to broaden out testing. One or two friends who haven't yet seen the game can sometimes be a huge help in offering a different perspective.
Questions
If we stop adding and changing things for a while, how do we feel?
Are we getting those "big moments" from gameplay where everything aligns?
What will committed players of this genre think about our current version?
Is something fundamentally wrong that's holding back the entire project?
What are the playstyles we think players will gravitate towards?
How can we reward those playstyles and will there be enough content for each of them?
"Internal" Alpha
Alpha is often best split into two main phases - I've put internal in quotation marks here as, for small teams, it's often going to involve others. If you do add more testers, they should still be sympathetic to your goals and ready to give constructive feedback that you can put to practical use.
At this point, your game should:
Be feature-rich, if not truly feature-complete
Have visuals which give a decent impression of the final look
Have enough content to start giving a true sense of progression
Have a very rough "first draft" balance pass
Your game will likely still be lacking most of its content and missing a fair amount of polish as well as quality-of-life features. It should, however, be very playable.
The purpose of this phase is to test all of your prior assumptions: you were certain during the Gut Check that things were ready to move forward, so were you right? This is your absolute last chance to alter core features, to rip up big parts of the game and start again with those if you need to - that's rarely the right call, but those rare instances are important to identify.
This phase is also about getting ready for a wider audience - the next time you test, you will have announced your game and therefore you will be dealing with the chaos of the general public! Make sure you relish the final shreds of your privacy and be honest with yourself about what you're going to unleash upon the world.
You'll need to build out a little more content and provide some more onboarding materials. I still don't recommend a full interactive tutorial just yet - it might be better to complete this phase, process the feedback and then build the tutorial immediately afterwards (more details on this to come) - but this is a situational judgement you will have to make. If you don't go for a tutorial, then carefully written text, images and video can all be helpful.
Questions
What's lacking from the long-term progression?
Where is the player going to end up eventually - is that still fun?
What does the player really need to know to get started?
What else do they need to progress?
How is that information best conveyed - in a linear tutorial format or contextually?
Do players have all the information they need from the UI or diegetic cues?
Is anything hard to see or hear - can the player miss important things?
Any remaining small features?
What will we need to eventually add in terms of polish - can we get some of that in now?
Can we now map out all of the content?
What do we need for a store page trailer that will really impress our target audience?
How much of the game can we get to that level of visual fidelity quickly?
Is optimal play fun?
Where do we need to get to in terms of balance, frustration and fairness?
Are we truly serving the different playstyles we identified earlier in development?
Tutorial Test
It can be wise to spend some specific time testing your onboarding, especially if you have been holding off on this up until now as I’ve suggested. Tutorials in general are an enormous topic - perhaps the best quick tip I can give is "be clear about standard mechanics; give space to anything unusual."
This is one area of game testing where in-person (or observed remote) tests can pay huge dividends: sometimes it's good to stop a test and discuss things with a player who is really struggling or has missed some critical information. Yes, you can cover all this as part of other testing phases but - as we'll see later - this can be very distracting from the important late-stage feedback you truly need.
If you're finding it difficult to set up an in-person test then consumer shows and universities can both provide excellent venues to get warm bodies in front of your game for this purpose. At a show, people are usually there with the express intention of trying new games and will give you a good sense of their initial reaction. University Game Development or Computer Science courses will often have willing volunteers, some of whom may be willing to put in a little more time - reaching out to staff members or course leaders is usually a good start. Many students love to hear about the practicalities of game development, so doing a talk or seminar in exchange for some testing time can be mutually beneficial trade.
One quick warning here: a very shallow game with loads of novelty will do great at a show but may not sell well in the long-term. Essentially, you've established a good Catch for that environment, but you have no idea of Hold - take it all with a pinch of salt.
Run a testing cycle with a small group first - you will have forgotten something obvious that causes an immediate problem. Fix that one thing, then send the game back in for a full cycle.
Questions
What's working about the onboarding and can we use that style elsewhere to improve player information?
Where are players getting stuck? What is the thing we keep wanting to say to those players to get them unstuck?
Is anything about our wording unclear?
Are there any common misconceptions among the players who are struggling?
Where is the tutorial itself getting boring, frustrating or patronising?
If that's only going to be true for some players, can we filter those players out by giving them the chance to progress faster or skip content?
Once the player has learned something new do they get a chance to practise it?
Do the default controls make sense? Is it clear to players how they can remap them if they don't like them (they can remap them, right??)
Is the early gameplay pitched at an appropriate level for a brand new player?
When tutorial text or V/O talks about an on-screen element, do we have a massive highlight around that element which is impossible to miss? (Sounds obvious but apparently isn't!)
When a player is looking for help on something are they able to find it? Are we using contextual tips for this?
Are there any secondary themes that emerge from watching many people go through the early part of the game for the first time?
External Alpha
This phase could well be your game's first true contact with the enemy (aka the real audience!) and so you will need to be ready for the inevitable onboarding headaches.
Hopefully, you've set up in advance by making sure that you have a small community around the game from announcement onwards. This is largely beyond our scope here but fundamentally involves:
Making as much of a splash with the announcement as you can
Giving the community somewhere to gather (often Discord)
Giving them reasons to stay engaged
You will now also have a road-tested tutorial (or equivalent contextual help) in your game ready to welcome brand new players. In a complicated game, there will still be elements that confuse people but the theory is that these should now be mostly under control.
Players know that this phase is a real indicator of how much developers care about their players. Without bending the project out of shape, serving that early audience by making things clear and then making it obvious that you are listening to their feedback can make or break a game.
Asking players to record video can also be a valuable exercise - with easily available software like OBS, it's possible for individual testers to capture their own gameplay and send it to you. For more outgoing players, they may also be able to record audio commentary, letting you know how they are feeling at key points.
Google Forms is a great way to quickly ask questions and collate the answers - there are several quintillion other survey tools out there though so just pick one that suits you.
If you want to get fancy with it, there are now also an array of playtesting services who can organise relevant gamer cohorts and use sophisticated platforms to automate the process of gathering video as well as gameplay telemetry. I would look into this route if your project requires a certain level of confidentiality, if you can't get direct access to your intended audience for some other reason or if you need a greater volume of high-contact testing than you're able to achieve on your own.
Ideally, you can run multiple alpha cycles, trying to make tangible improvements each time, bringing in new cohorts and continuing to work with longer-term players. You can also do a "live service" setup where you're able to quickly push fixes and respond to issues in realtime - on certain types of game this can be an ideal scenario, as keeping your current players engaged is vital.
You now get to ask your players direct questions: you can stop guessing about how things will be received and actually know. This is why you want to be very clear on your game's fundamentals and onboarding prior to this point, as you can spend your time here focussing on content, progression and balancing. In the modern gaming environment, this is the stuff that really counts - plenty of cool games are torpedoed by a lack of content or an unsatisfying stale meta.
Finally, this is a great chance for any remaining red flags to emerge - if your testing has missed something in an earlier phase, you may well still have time to react before it's too late.
The following questions are drawn from a pool that I’ve used in past public tests - there’s nothing particularly special about them so - as ever - please adapt as you see fit.
Questions
How much fun did you have with the game?
How many sessions did you play before responding to this questionnaire?
Please give us your general impressions of the game! More specific questions will follow but this is a chance for you to just give us a general idea of how your playtest went - write this as you would a Steam review.
Did you feel that you were consistently making interesting decisions while playing?
Was there anything you found too obvious, repetitive or too random?
From memory, describe a sequence of gameplay you really enjoyed.
Did you feel like you were able to consciously use strategies that suited the way you wanted to play the game?
What is, in your opinion, the worst aspect of the game?
Did you have any other serious problems with it?
What would you like to see more of?
How did you find the game's difficulty?
How fair was it?
Do you think the game is balanced well?
Does anything seem significantly over- or under- powered?
What caused you to end a session - did the game get boring, fatiguing or frustrating at any point?
Was there anything you found confusing or too hard to execute?
How much would you currently pay for this game?
What features or content do you feel it would need for you to pay $5 more?
Did you encounter any serious bugs, crashes or other issues?
Anything else you would like us to know about the game or feel should be improved?
External Beta
This is a chance to figure out your roadmap to release and to test final assumptions. Again, this is best done over many cycles and updates - many games benefit from an extended final polish period.
You can go broad with betas - indie developers often worry about allowing too many people to play the game too early, but in most cases you won't reach even a tiny fragment of your total market, so bring in as many players as you think will be manageable. Your objective here is essentially "front-run anything which might come up in a Steam review" - you are reaching into the future, bringing back knowledge and then acting on it in advance.
The emphasis should now be much more focussed on final content, polish, UX and bugs but ignore more fundamental feedback at your peril. Some games, such as large-scale multiplayer or competitive titles, will require a lengthy public beta to get balance into the realm of acceptability for a 1.0 launch - however long your beta, you need to make sure that you keep up regular community contact.
You may not need to continue with video testing at this point but questionnaires and chatting with players on a platform like Discord are still a good idea. Also, don't be afraid to ask basic questions (or repeat questions from previous phases) and perform final checks on your onboarding, particularly with new cohorts.
Difficulty tuning may also be slightly too weighted towards experienced players in the early game at this point - this insightful piece from the developer of the exceptional Highfleet has some good thoughts there. You want to make sure both long-term players (hopefully you retained them from your alpha - if not, you may have serious retention issues to be addressed!) and brand new players are having a great experience.
In theatre, there is often a separate "tech" and dress rehearsal - it's sometimes said that you want everything to go wrong in the tech and everything to go right in the dress. A disastrous tech means that you become aware of every little potential problem, and you can develop solutions and strategies before anything goes in front of the audience. Subsequently, big showstopper problems can take a back seat in the dress, and you can focus on anything which still stands in the way of success - that's the theory anyway! Similarly in games, having things fall over early in the beta phase is great - the more that goes wrong now, the fewer surprises may happen later.
In an ideal world, your release candidate should be locked with a good buffer zone to release. Sometimes, developers do not get that luxury and if you find yourself up against it be sure to allow time for basic checks.
Before you release your game, have everyone on the team play through the release candidate again one final time. Be certain.
Your playtesting questions at this point should really be a "greatest hits" of those from previous phases. Reinforce the critical elements; challenge your assumptions and try to avoid any nasty surprises on launch day.
Soft Launch, Early Access and Metrics
Most mobile games go through a soft launch in a low priority territory in order to start gathering metrics. In a similar vein, some PC indies like to launch a version on Itch or in another limited form.
This isn’t an essential phase for many games, and you should be getting high quality information from your beta, but paying customers who are truly expecting a finished product will always exhibit different behaviours and hold specific opinions. Also, it’s often at this point where data can start to take over - there’s nothing to stop you implementing analytics earlier, but it’s often considered to be best to do this with live paying players.
Most metrics advice tends to center around identifying specific cohorts, tracking their retention over variable windows (often D1, D7, D130, D90, D180 etc where D is “day”) then figuring out a retention profile from there (if you want to get very deep into the technical detail of this and implement your own analytics system, take a look at Game Analytics by Russell Ovans). There are plenty of off-the-shelf solutions available for actually implementing analytics, but it’s important to understand what you are doing with them.
Often, it’s a good idea to frame a yes / no or A/B style question, implement the changes and observe the accompanying shift in analytics. Occasionally, analytics will flag that players are dropping off at a particular point in the game, or quitting in frustration. Also, if your implementation is thorough enough, you may be able to get useful balance data - are there any obvious trends in winning or losing player strategies for example?
The advantage with analytics is that they reflect what players are actually doing rather than what they are saying - I was once told by a senior developer working on a popular mobile game that their results dramatically improved when they started completely ignoring player comments on the forum and just focussing entirely on metrics! While I don’t advocate this approach, particularly for smaller indie games, using data to observe behaviour that players may not be able to fully articulate or identify might be a more productive framing.
If you are looking into Early Access on PC, you should be aware that this is not directly analogous to a soft launch. This is another enormous topic which I will cover in more detail at some point, but essentially players are expecting a solidly playable game with a good amount of content - the ideal Early Access build is:
Visually polished, though not 100% visually complete
Free of showstopper bugs and crashes, with limited “high irritation” bugs
Offering a strong amount of replayable content, enough for players to play for several weeks before expecting an update
Early Access games are a product - players expect a regular, high-communication update cadence where you call your shot and hit it every time. If you can achieve that and your initial build is impressive, you can use Early Access to continue gathering testing data, however your focus should really be almost entirely on adding content. This type of launch suits highly systemic games with a lot of emergent gameplay, and those where the content pipeline is reliable and fully mapped out - other styles of game may well struggle.
Interpreting Feedback
One ancient bit of wisdom that game developers like to repeat to each other is "players can identify problems but not solutions". This is good advice for beginner devs who can sometimes chase their own tail with player feedback, but when applied globally it is both wildly reductive and sometimes profoundly wrong.
In fact, players vary widely in their ability to identify true problems or propose valid solutions. Sometimes an individual will nail both the problem and the solution in one fell swoop; sometimes they'll be obsessed with an issue which doesn't matter at all.
You want to read feedback critically, looking for themes and trends in context. Yes, maybe some players find the pace too slow but they may be from the wrong cohort; maybe the first level is too easy for players who have thousands of hours in the biggest game in your genre etc. Sometimes, it is correct to ignore feedback. If you struggle with this, hire someone with an arts education (turns out we do have economic utility after all)!
“During pre-release testing, [The Legend of Zelda] players became lost among the maze-like dungeons, and grumbled about how confusing it all was. Miyamoto bravely responded by making the game slightly more difficult; he removed the sword from Link’s inventory at the start, and gave players the additional task of locating it. The result, he hoped, would be a game that forced players to communicate with one another in order to solve its mysteries."
Ryan Lambie - Den of Geek - https://www.denofgeek.com/games/the-inspiration-behind-the-legend-of-zelda/
One area that requires particular care is player wishes and fantasy features - you want the "wouldn't it be cool if I could…" feedback very early in the process, but good games unfortunately invite a lot of this daydreaming at a late stage. Keep it, mull it over, think about it for DLC or free updates but don't let it distract you from shipping.
Above all else, you want to align feedback with the research you've done around your chosen genre and playerbase, as well as with your own instincts as a designer. Right or wrong, your game is yours - if you are honest with how you handle input, you will be able to learn valuable lessons for your next go around the block.
Test Complete
Testing is a tool to help make your game development better. It is a chance to engage with real people, in all their complexity and contradiction - learning to interpret their behaviour and feedback is one of the most important skills for a designer. You can choose what you do with that feedback - it should enhance rather than compromise your vision. If you can gain a greater understanding of the effect your work is having, you can turn that to your advantage both creatively and commercially.