Three Questions You Shouldn’t Ask During User Research

by Jared M. Spool

The participant was struggling. While he was a high-volume customer who had bought tons from this site in the past, today he wasn’t getting along with the checkout process. Confusion happened in both directions: the shopper didn’t understand what the site was trying to tell him and the site definitely didn’t understand what he wanted.

The team spent several painful minutes watching as this poor customer made slow progress. His determination made the purchase happen, but it wasn’t smooth.

“It’s usually not this bad,” the customer shared. “I’ve had problems before, but this feels especially difficult today.” It was just a perfect storm of things-not-going-well.

Sitting next to me was a team member who was becoming increasingly anxious. As the e-commerce site’s VP of Customer Service, he could feel this customer’s pain. This wasn’t the experience he wanted their customers to have, especially a loyal, regular customer like this one.

At the appropriate moment, the session’s moderator turned to the observers and asked, “Does anyone have any questions?”

“Yes, I do,” exploded the VP. “Do you think… I mean, if you were at home and you went through all this and had all this trouble… and there wasn’t a room full of people watching you… do you think you would’ve called the 800 number that was on every screen? Would you have called our support department?”

The VP wanted to believe that, when customers have a traumatic experience like this, they don’t just let it go—they call his people for help. After all, he’s made sure they’re trained to handle exactly this type of problem and shower the customer with great attention and results. The guy has to want to call.

The customer thought about it carefully for a few moments. You could almost see into his mind as he recreated the situation in the comfort of his own home. Finally, he said, in a very calm voice, “I think so. Yes. I would’ve called the 800 number.”

The VP sat back in his chair, happy with the thought his team would be there for the rescue.

Then I asked, “Have you ever called the support number when you’ve tried to purchase things in the past?”

“No, I haven’t,” the customer answered immediately.

That’s the puzzle: The customer said he would call support, yet also said he had never done so in the past, even though he’s had similar problems. Which should we believe?

1) Don’t Do: Asking About the Future

Unknowingly, the VP broke one of the rules of good participant questions: Don’t ask about the future.

When asked about a future scenario, many people will try to answer honestly. They’ll answer about what they’d like as their ideal behavior. They want to think they’ll behave in the most logical, effective manner. In short, they want to think they’ll do the right thing.

Yet, in the thick of the situation, people aren’t always so logical. They don’t do what’s ideal.

We want to predict future behaviors, so we can make designs that service them. Yet just asking participants may not get the actual answer.

I’ve found the best way to predict future behavior is to look at past behavior. Asking a participant about what they’ve done in the past is a better way to get those answer.

Instead of asking, “do you think you might need to store messages in different folders?”, I’ve found it better to ask, “when you’re done with messages, what do you do with them?” I focus on what they’ve done in the past, looking for that behavior to tell me what makes sense.

Asking about the future is just one type of question I’ve learned we shouldn’t ask our research participants.

2) Don’t Do: Asking How They’d Design a Feature

“If we gave you a way to annotate your files, how would that work?” “What would be the right way to organize the menu options?” “What other fields should we include on this form?” “How would you want to see this data displayed?”

I’ve heard these questions many times. (I’ve even asked them on occasion.) After all, the designers want to create an effective design for the user. The best way is to just ask the participant what that design should be, right? Wrong.

Users aren’t designers. (If they were, they wouldn’t need us.) They don’t know how to deal with constraints. They haven’t really thought the problem through.

Any answer they give us is unlikely to be a great design. It’ll be the first thing they think of and, unfortunately, they’ll take a long time to get there.

I saw this happen just the other day. A smart team was talking with a long-time customer of their product. “How would you…” the team leader asked.

The customer’s posture changed completely. Whereas before they’d been leaning towards the team members with solid eye contact for every interaction, they now relaxed in their chair and looked into space. “Hmmm. I guess I’d…” as they tried to imagine their brilliant idea.

Except, it wasn’t a brilliant idea. It was simplistic and couldn’t handle the myriad of edge conditions that were patently obvious.

The team could see the original problem and asking how to design it wasn’t adding anything. They knew what to do.

3) Don’t Do: Asking by Providing Their Reason

“Is the reason you don’t click this button because it’s really hard to see?”

This is a tough one for many observers. We’re watching the participant and they aren’t doing what we know to be the optimal method. Our brain races with possible reasons. Need to find out. Must ask.

“Why didn’t you click this button?” is only a hair better. It doesn’t telegraph the reasons we’re hypothesizing, but it implies the person has done something wrong. They’ll tell you a reason, but it may be more because you’ve pointed it out, not because it’s what they were actually thinking.

Be creative about exploring the participant’s understanding of the interface. For example, when I’m curious about why they didn’t click a particular button, I ask them to talk about what each button does. That way, they won’t associate the specific button with the action they just did (and I learn their overall understanding of the controls).

One Bad Question Won’t End The World

Asking these questions isn’t evil. We’re not violating some Geneva-Convention-type of international agreement. We won’t be hauled away by the User Research Police.

However, there is a price to asking the wrong questions. When conducting user research, the most valuable moments are limited times that the team spends with each participant. It’s important to make every second count.

Asking one of these questions wastes those precious moments. They take time without returning value. The participant tries (and, usually tries really hard), but that trying doesn’t pay off and eats down the clock.

Learning to focus on the right questions can help you get the most out of each session.

Published here on August 18, 2010.