Sluggy wrote:
3) You probably already know this but if a picture is worth a thousand words, then a video stream is worth billions. If you can convince someone play the game while you silently observe you'll get so so much more information than any questionnaire could provide. If you can record the play sessions that will help a lot too because you'll likely miss many details the first time, plus being able to re-watch certain clips will allow you to break down user actions and deduce what kinds of things you can/should/need to change.
A good idea, but with some technical effort you can do even better: in a alpha or beta version you can automatically record input replays of every single game and have them sent to you, instead of recording a few huge video files, receiving them only if everything goes well, and potentially wasting time watching them.
Using replays, more information than what can be seen in a video is available, and you should be able to automate some important data analysis.
Simulating game states throughout a game allows answering statistical questions like
- Do players have the expected stock of powerups, bombs etc. when they reach some milestones, like bosses? If they have less, were the missing items lost after dying, expended early or not picked up? If they have more, where is the exploit?
- Where in the level do players get killed most often? By what possibly unfair enemy attacks?
- Are the players attacking enemies with the "right" weapons?
- Which paths, in case the next level can be selected by the player like in Gradius or Deathsmiles, are more popular? Which ones are associated with higher or lower scores or with significant events (e.g. running out of bombs)?
This way you could watch only a few playthroughs in full, for inspiration and clarification, rather than to collect raw data. You know what replays contain interesting occurrence, and when.