Get Stuck Through Someone Else’s Eyes

Posted by admin on July 29th, 2013 filed in Development, NetBeans

I once had the opportunity to sit behind a one-way mirror once during a user study for NetBeans. It’s eye-opening to see the reactions and interpretations of users who don’t have the same presumptions that I have.

We asked the subjects (all developers or CS students) to pretend their boss had given them the task to evaluate a new development tool, NetBeans. Since we wanted to evaluate the quality of our web site, we also included a direct link to netbeans.org. I don’t recall the literal questions, but it was something like: “What would you tell your boss what this development tool can do?”, and more specifically, “Your goal is to develop an XYZ application. What can NetBeans do for you?”

I came across this article (https://articles.uie.com/three_questions_not_to_ask/) and I think our usability team did a good job those days. The article has three recommendations:

Don’t ask users leading questions about the future such as “Surely, you would call the help line now? Surely, you would now use the search?” You only get relevant responses if you ask open questions about the past: “Did you have similar issues before? What did you do?” Maybe the response is “Last time, I gave up and used < the competitor's product >”

There’s a saying, if someone like Ford had asked people what they wanted, we’d now have faster horses, not cars. Talking to users is important, but don’t ask them “How would you design this?” — that’s not their job, it’s yours! Ask them what their goals are (“to travel from A to B”) and what they want to keep (“independence and speed”) and what they could do without (“stepping in horse manure while walking!”), and work with that.

Don’t impose your expectations by asking questions that imply that the user is doing it “wrong” by not following your script. This way you won’t gather new information. The article mentions negative examples such as “Was the button too hard to see?” and “Why did you not click that button?” Rather, ask open questions such as “Describe in your own words what this application can do? Describe what each button does?” to discover what your users expected.

When doing a user study, don’t create an echo chamber of what you already know. You want to see the application through the users’ eyes.

Comments are closed.