Good feedback questions for level testers?

A place for people with an interest in developing new shmups.
Post Reply
Iori Branford
Posts: 328
Joined: Sat Apr 15, 2006 5:47 am
Contact:

Good feedback questions for level testers?

Post by Iori Branford »

I'm thinking I should start having my levels playtested. I would post test builds in trusted forums and anyone could test at their leisure and fill out a feedback form.

Questions so far:
  • # of times level played vs # of times level cleared
  • Prior shmup experience level
  • I stopped playing because...

    Agree or disagree:
  • I could clearly make out enemies and their attacks, including those attacking from the flank or the rear, and some ways to avoid them
  • Enemy attack intensity was appropriate for this point in the campaign (_th level out of a planned _)
  • I had the right amount of resources (options, bombs, etc) to fight the enemy, without starting on a previous level
Any other good questions? Any of these not so good? How many questions is too many?

Preview:
https://goo.gl/forms/x9fSLB0BQwgqTACh1
Sluggy
Posts: 12
Joined: Mon Jul 16, 2018 10:47 am

Re: Good feedback questions for level testers?

Post by Sluggy »

Just wanted to make a couple suggestions based on my own experience doing this sort of thing.

1) The question 'Prior shmup experience level' isn't what I would consider well formed. It's too open ended, vague, and somewhat meaningless. Something more like 'How often do you play games like this one?' along with provided cues like 'All the time', 'Occasionally', 'Almost Never' would be easier for people to answer and for you to comprehend the result.

2) I certainly understand your desire to get a lot of questions answered but avoid the temptation to throw the whole list at every player that comes along! There are a couple of reasons for this. a) Nobody actually wants to answer your questions. Like, ever. They might be keen to give opinions and make comments but filling out what feels like a test just isn't fun. b) You want to have focus. Pick one aspect that you are specifically curious about and then cull your list of questions down to the most important ones related to it. For example: if you want to know how players feel about item drops, ask them things like, 'Do weapons drop often enough?', 'Should they drop more often?', 'Are they interesting?', 'Which was your favorite?', 'Least favorite?' You don't want to overload their thoughts with a bunch of seemingly unrelated topics and if you keep it focused you'll likely find they are more tuned in to that after thinking about it just a little and are now ready to give more detailed information in that space you've inevitably left as 'Additional comments and thoughts are welcome here!'

3) You probably already know this but if a picture is worth a thousand words, then a video stream is worth billions. If you can convince someone play the game while you silently observe you'll get so so much more information than any questionnaire could provide. If you can record the play sessions that will help a lot too because you'll likely miss many details the first time, plus being able to re-watch certain clips will allow you to break down user actions and deduce what kinds of things you can/should/need to change.

Best of luck!
Ixmucane2
Posts: 760
Joined: Mon Jan 19, 2009 3:26 pm
Location: stuck at the continue prompt

Re: Good feedback questions for level testers?

Post by Ixmucane2 »

Sluggy wrote: 3) You probably already know this but if a picture is worth a thousand words, then a video stream is worth billions. If you can convince someone play the game while you silently observe you'll get so so much more information than any questionnaire could provide. If you can record the play sessions that will help a lot too because you'll likely miss many details the first time, plus being able to re-watch certain clips will allow you to break down user actions and deduce what kinds of things you can/should/need to change.
A good idea, but with some technical effort you can do even better: in a alpha or beta version you can automatically record input replays of every single game and have them sent to you, instead of recording a few huge video files, receiving them only if everything goes well, and potentially wasting time watching them.
Using replays, more information than what can be seen in a video is available, and you should be able to automate some important data analysis.
Simulating game states throughout a game allows answering statistical questions like
  • Do players have the expected stock of powerups, bombs etc. when they reach some milestones, like bosses? If they have less, were the missing items lost after dying, expended early or not picked up? If they have more, where is the exploit?
  • Where in the level do players get killed most often? By what possibly unfair enemy attacks?
  • Are the players attacking enemies with the "right" weapons?
  • Which paths, in case the next level can be selected by the player like in Gradius or Deathsmiles, are more popular? Which ones are associated with higher or lower scores or with significant events (e.g. running out of bombs)?
This way you could watch only a few playthroughs in full, for inspiration and clarification, rather than to collect raw data. You know what replays contain interesting occurrence, and when.
Sluggy
Posts: 12
Joined: Mon Jul 16, 2018 10:47 am

Re: Good feedback questions for level testers?

Post by Sluggy »

Ixmucane2, definitely a good idea! Though I'd argue it's not necessarily better, just a different toolset. To paraphrase the developer of Anti-Chamber, 'Analytics doesn't measure player frustration.' People's actual emotional reactions to your game, good or bad, can't be replaced by statistics.
Post Reply