Sandra and I both come from the non-casual games industry, so our idea of a QA checklist is very heavily flavored by that experience. I don’t know what other casual game developers use, but here’s how we made our list.
- By necessity, we needed to roll balance-testing and bug-testing into the same QA sessions. (Of course, at some point, you have to declare balance-testing “done” and then only do bug-testing.) So the QA checklist needed to be granular enough to support useful comments of either type.
- We’re too small to bother with a bug database; instead, bugs can be reported right on our QA sheet.
- We only have two QA people, so we used a table on a wiki page. One column was for my comments, and the other column was for Sandra’s.
- We repeat the QA checklist again and again with new builds until everything comes out just right.
- This QA checklist was for very late in the process; we’d done unit-tests before this, as well as a full beta cycle. So some�minor tests weren’t on this final list.
So, here’s our QA test list. Imagine that this is a wiki table that either person can edit. When the sheet is full, we make a new build with any needed adjustments, and try it all again. We know when we’re done when both people have entered “OK” or some other comment for each entry, and no further builds are forthcoming.
Starcrossed QA Test List
With a release build on a clean install (delete all game registry keys, too!) test the following:
- General Considerations:
- Can you safely left-click, right-click, and double-click all buttons in the game?
- Does the user name look fine on all GUI screens when using widest-possible name (lots of Ws)?
- Start an untimed game from each of the Resume Points, play a full level
- Quit game during play and resume (make sure auto-save works)
- Quit game during a cut-scene and resume (make sure auto-save works)
- Quit game during a pop-up tip and resume (make sure auto-save works)
- Exit game to main menu, switch between menus rapidly, and return to game (x10)
- Press Alt+Enter repeatedly to switch back and forth between windowed mode and full screen.
- Press Alt+Tab repeatedly to switch from full-screen app to some other app.
- Does the game look correct on wide-screen monitors in full-screen mode?
- Make sure cheat code #1 works as intended and can’t break anything in-game
- Make sure cheat code #2 works as intended and can’t break anything in-game
- Make sure cheat code #3 works as intended and can’t break anything in-game
- Verify Account Screen Works Correctly:
- Create and delete an account. Does it leave tracks in registry?
- Try creating 50 accounts. Everything fine?
- Try deleting all 50 accounts, including the currently active one. Everything behave correctly?
- Try creating accounts with the same name as existing accounts. Correct behavior?
- Try creating accounts with illegal characters / no characters / too many characters. Blocked correctly?
- Try creating account names with spaces before or after the name. Cleaned up correctly?
- Verify Options Panel Works Correctly:
- Slide knobs up/down (then hit OK between changes) and make sure changes take place
- Quit game while viewing the Options screen. Any problems?
- Make sure changes are saved between game sessions
- Test the “3D support” box; play a game with/without this checked
- Verify Tutorial Screen Works Correctly:
- Does it look correct?
- Quit game in mid-tutorial screen. Any problems?
- Verify that “Don’t show this again” checkbox works
- Verify High Score Screen Works as Intended Regardless of How You Get To It:
- Are relevant tips displayed?
- Is the proper score screen shown (when arriving from a game mode, show that game mode’s score; when arriving from main menu, show the previous game mode’s score, or Untimed mode if no game)
- Is the last game’s score highlighted?
- Do scores move down appropriately when new higher scores are added above them?
- Does the high-score list look fine with entries that are the widest-possible name (lots of Ws)?
- Does the high-score list look fine with names of varying sizes and shapes?
- Verify Data:
- Check for usage.log [our log file]. Shouldn’t exist in final version!
- Check readme.txt for up-to-dateness
- Make sure install bundle does not include inappropriate files (.PSP files, raw .TGAs, etc.) left over from development sessions
- Verify all tips.txt are valid and useful, and are in the right categories
- Untimed Game Mode
- Play�game to end-screen.
- Play an entire game in ones session, start to finish.
- Does beating the game unlock Mastermind mode?
- Verify highest-level-reached is stored correctly even if game is not completed properly
- Story screens make sense? Right order?
- Music changes correctly?
- Notes on sound issues:
- Pop-up hints useful and appropriate?
- Balance/Bug Notes on Level 1:
- Balance/Bug Notes on Level 2:
- Balance/Bug Notes on Level 3:
- [etc. up to level 33]
- Timed Game Mode
- …
And so on. The list goes on and on, with similar sections for all six of the game’s modes. We were pretty confident with the game at this point (it went through beta testing with hundreds of users and no major bug sightings), so this is really just the final-stage pass. Even so, we’ve discovered several bugs and tons of balance issues.
In the end, you just can’t skimp on the gameplay time. The QA list intentionally requires us to play through the game many times to make sure we see lots of random game situations. Games with lots of randomness need lots more face-time to find unusual cases.
People don’t seem to talk about the QA process very much, but it’s pretty important. Few things sour a game’s presentation more than a sudden crash or an obvious glitch.