Don't do what your users say ...

Apr-16-07, 11:49 am by Hanford
File under: Video games, design, User Interface, game design

... do what they're telling you. This came up at GDC when talking to the Mawsoft guys so I thought I'd blog about it here.

Community is big these days. You'll hear lots of designers tell you that it's important to build a strong community and listen to them, because they are your core users. And I agree with that.

But in UI design it's important to understand that what a user says and what a user is telling you can be two different things. It is rare that a user outright lies for no reason. There is almost always a root cause for what your users are saying. The trick is to find that root issue to truly get what the user is telling you. And it is often a bit different than what their words are saying.

An example

My first solo-effort game was a little one called Rocknor's Bad Day, which was mainly a learning experience for me. In the game you control a little robot named Rocknor, and the goal is to make it through each level and get to the exit. Along the way you are confronted by many obstacles, puzzles, and traps.

I sent a private beta version to some close friends to get their take on it. A few days later I collected feedback via email and phone conversations. I got a good variety of comments back. Constructive thoughts. But I noticed an interesting trend: The most common thing suggested was "Add an undo to the game". It seems almost everyone who tested the game had asked for an undo option.

I was kind of crestfallen, for a few reasons. Games aren't the type of products that have or need undo. word processors and paint programs Yes, but not games. Secondly, retrofitting an undo into my game would be a bucketload of work, if I was even up to the task of doing it.

Watching users play

My first natural thought was to be a good designer and take my user's suggestions to heart. But I decided to do a deeper test of the game before venturing off into the world of undo programming. I wanted to find the root cause of the "undo" request. I had some friends of mine host a playtest party at their house, where everyone brings laptops and installs the game, then I could watch them play it. Not quite a genuine user test, but more of a Lucasarts-style Pizza Orgy.

During the party I got a lot of great feedback. Just watching someone play my game and see them learn from their mistakes was an incredible experience. But mainly I was watching closely to see if and why anyone was going to request an undo feature. What I saw was surprising. Almost everyone playing the game was oversteering the character. I mean really oversteering. Overshooting their targets, pushing objects too far, smashing into walls, and falling into water.

One of the major elements of Rocknor's Bad Day is pushing obstacles into place, and the obstacles can only be pushed, not pulled. But oversteering of the main character meant that they were often pushing objects too far into unrecoverable positions -- which means they lost the level and would have to restart. I also saw a lot of people often fall into the water way too often, which was instant death.

The results

After the user test is was clear to me that the root cause for undo requests was that the controls were too sensitive for the average player. There were a few other things that were revealed too. People really loved solving the puzzles in the game -- the first time. But if they had to restart, they really did not enjoy redoing the puzzles they had already solved. This was another cause of wanting an undo in the game. I ended up making a few changes:

I slowed down the movement of the character
When he was pushing obstacles he moved even slower
There was no real gameplay benefit to having water be a death obstacle, so I made water act like a wall to the player -- meaning he would not fall in, but he couldn't walk on it or in it, either.
I implemented waypoints that, once the player had touched, would be their respawning point if they restarted. Waypoints were often located on the other side of difficult puzzles so if the player restarted they would not have to re-play the already solved puzzles again.

Post-change testing

After making the changes, I did a few more user tests and the results were astounding. No one was asking for Undo anymore. It was a big difference from before. The important thing though is that I never implemented the number one feature request, which was undo. I didn't ignore their requests, but I didn't go ahead and implement them without understanding the root cause of their requests. When you listen to what your users are telling you instead of what they're saying, you have the opportunity to incorporate improvements that still fit into your vision of your product.

17 Comments | Retweet this

Feedback - 17 responses

Displayed newest to oldest. Leave a comment.

Torley wrote: Sep-16-08, 6:44 am

Oh my gosh, that Rocknor screenshot has lots of neon watermelon colors, green + pink! My fave! :D

Kyle Pontier wrote: Aug-26-08, 9:49 am

I have found that there is usually a discrepancy between what users "say" and what users "do". It is important to listen to what users are “saying”, but more importantly... understanding what users are “doing” in a contextual setting is much more valuable for design decisions.

JR wrote: Aug-13-08, 4:16 am

This is a lesson from systems (and software) engineering. Users will say all sorts of things; understanding their needs and turning them into requirements that can be implemented by a developer is what separates satisfactory design from great design.

Also: bravo to you for play-testing and eagerly collecting feedback from the users. Getting rid of pride ("They don't know what they're talking about, I've obviously done it right already!") is the first step to improving your work. Nice job!

Mike G wrote: Aug-11-08, 2:20 pm

Great post! Bookmarked for future use as evidence at work :-)

Dave Shuck wrote: Aug-11-08, 12:04 pm

Unfortunately the art of gleaning what the user is actually telling you is often as much of an art as writing the solution!

Mike wrote: Aug-11-08, 11:08 am

Excellent article and it can be applied to solving all kinds of problems, not just computer games.

Moral of the story; find the root of the problem.

Tom Ritchford wrote: Aug-11-08, 5:55 am

Nice, short, well-written and I agree with the argument completely. I also write music and I observe the same phenomenon - I listen to everyone's criticisms very carefully but often I ignore the specifics and say, "What they *really* want is..."

(In particular, when they say, "We need more X" I always read it as "Turn down Y which is getting in X's way" - so I hear "more cowbell" as "less rhythm guitar", so to speak.)

stewie wrote: Aug-11-08, 5:25 am

What I got out of this was that users and developers speak a different language (the users not being technical experts). There's still the hint of "blame the user" bias to your post, but it is pretty much on target once you realize the obvious miscommunication. We need to heed the users' requests, but we also need to translate their layman terminology into the appropriate code.

Jim McDosh wrote: Aug-11-08, 4:22 am

I think user inout is very important.

JT
www.FireMe.To/udi

MM Eternal wrote: Nov-24-07, 7:59 pm

I dont really have many complaints about either the classic or the new version of MaidMarian in the sense of playing and graduating to higher levels etc. However I do take exception to players not playing and fighting fairly (backstab, hit u when ur are down, etc., Additonally the dating scene chat is not what the game is intended in my belief (other than making cordial friendships). The potty mouth, spamming, website tricks that are really very filthy porn sites, need to be left out of the game. Furthermore, the use of hacks and cheats is very annoying, especially to someone like my self who spends many hours gaining my exp and weapons etc. the right way.

Keep up the good work!

MM Eternal

axcho wrote: Jul-6-07, 8:39 am

Very true. :) But I wish it were easier to figure out what they're telling you from what they say!

The most common complaint I've gotten about Braids is that the controls are bad. I know the main problem is that they are hard to learn, but I don't know yet if there is a deeper issue than that.

Hanford wrote: Apr-24-07, 12:17 pm

The funny thing is looking back, there's still a lot of feedback that I didn't fully understand, and when I play the game today I see all sorts of things I'd change. Rocknor's Bad Day was a real learning experience.

Marty Plumbo wrote: Apr-24-07, 12:06 pm

I agree, Hanford, et al. Players/testers will generally articulate themselves in terms of what they think needs to be added/removed/fixed: "Get rid of X", "Add a Y". In reality though, you should generally filter their comments to hear what isn't working but then come up with the solutions yourself. After all, you're the game designer.

I made a similar mistake once by assuming that my testers would "get" an unconventional play mechanic that I was developing (http://martyplumbo.com/?p=32). They didn't. Instead of evaluating the new approach, most just told me how to "fix" the game and make it more like various games that they were already familiar with.

maninalift wrote: Apr-24-07, 12:46 am

Yes, verily. UI by democracy can never work. This also relates to the basic task of finding the simplest way to implement all of the desired functionality in one interface rather than just piling feature on feature.

Adam wrote: Apr-22-07, 7:30 pm

Wow, on the mark. I'm gonna show this to a UI friend, who will undoubtedly approve of the idea and example as well.

This also highlights the difference between developers / companies who follow (what's "hot" for users today) vs. those who lead (innovating new user interfaces, new ways of looking at things, completing tasks, etc.). The latter: more risk, much more potential to gain :).

Hanford wrote: Apr-17-07, 1:58 pm

Yeah, you were totally on-board!

RohoMech wrote: Apr-17-07, 1:28 pm

I *think* I agreed with you during GDC, but your post crystalizes your point much more, so nice post :-) Hopefully people will listen to what you're saying.

Comments are displayed on posts and visible to all site visitors.