Menu

Beta — We’re reformatting our articles to work with the new site, but this is one hasn’t been done yet; it may contain formatting issues. No need to report this. We’re working on it.

Three Questions You Shouldn’t Ask During User Research

by Jared M. Spool
on August 18, 2010

The participant was struggling. While he was a high-volume customer
who had bought tons from this site in the past, today he wasn’t
getting along with the checkout process. Confusion happened in both
directions: the shopper didn’t understand what the site was trying
to tell him and the site definitely didn’t understand what he
wanted.

The team spent several painful minutes watching as this poor
customer made slow progress. His determination made the purchase
happen, but it wasn’t smooth.

“It’s usually not this bad,” the customer shared. “I’ve had problems
before, but this feels especially difficult today.” It was just a
perfect storm of things-not-going-well.

Sitting next to me was a team member who was becoming increasingly anxious.
As the e-commerce site’s VP of Customer Service, he could feel this
customer’s pain. This wasn’t the experience he wanted their
customers to have, especially a loyal, regular customer like this
one.

At the appropriate moment, the session’s moderator turned to the
observers and asked, “Does anyone have any questions?”

“Yes, I do,” exploded the VP. “Do you think… I mean, if you were
at home and you went through all this and had all this trouble…
and there wasn’t a room full of people watching you… do you think
you would’ve called the 800 number that was on every screen? Would
you have called our support department?”

The VP wanted to believe that, when customers have a traumatic
experience like this, they don’t just let it go—they call his
people for help. After all, he’s made sure they’re trained to handle
exactly this type of problem and shower the customer with great
attention and results. The guy has to want to call.

The customer thought about it carefully for a few moments. You could
almost see into his mind as he recreated the situation in the
comfort of his own home. Finally, he said, in a very calm voice, “I
think so. Yes. I would’ve called the 800 number.”

The VP sat back in his chair, happy with the thought his team would
be there for the rescue.

Then I asked, “Have you ever called the support number when you’ve
tried to purchase things in the past?”

“No, I haven’t,” the customer answered immediately.

That’s the puzzle: The customer said he would call support, yet also
said he had never done so in the past, even though he’s had similar
problems. Which should we believe?

1) Don’t Do: Asking About the Future

Unknowingly, the VP broke one of the rules of good participant
questions: Don’t ask about the future.

When asked about a future scenario, many people will try to answer
honestly. They’ll answer about what they’d like as their ideal
behavior. They want to think they’ll behave in the most logical,
effective manner. In short, they want to think they’ll do the right
thing.

Yet, in the thick of the situation, people aren’t always so logical.
They don’t do what’s ideal.

We want to predict future behaviors, so we can make designs that
service them. Yet just asking participants may not get the actual
answer.

I’ve found the best way to predict future behavior is to look at
past behavior. Asking a participant about what they’ve done in the past
is a better way to get those answer.

Instead of asking, “do you think you might need to store messages in
different folders?”, I’ve found it better to ask, “when you’re done
with messages, what do you do with them?” I focus on what they’ve
done in the past, looking for that behavior to tell me what makes
sense.

Asking about the future is just one type of question I’ve learned we
shouldn’t ask our research participants.

2) Don’t Do: Asking How They’d Design a Feature

“If we gave you a way to annotate your files, how would that work?”
“What would be the right way to organize the menu options?” “What
other fields should we include on this form?” “How would you want to
see this data displayed?”

I’ve heard these questions many times. (I’ve even asked them on
occasion.) After all, the designers want to create an effective
design for the user. The best way is to just ask the participant
what that design should be, right? Wrong.

Users aren’t designers. (If they were, they wouldn’t need us.) They
don’t know how to deal with constraints. They haven’t really thought
the problem through.

Any answer they give us is unlikely to be a great design. It’ll be
the first thing they think of and, unfortunately, they’ll take a
long time to get there.

I saw this happen just the other day. A smart team was talking with
a long-time customer of their product. “How would you…” the team
leader asked.

The customer’s posture changed completely. Whereas before they’d
been leaning towards the team members with solid eye contact for
every interaction, they now relaxed in their chair and looked into
space. “Hmmm. I guess I’d…” as they tried to imagine their
brilliant idea.

Except, it wasn’t a brilliant idea. It was simplistic and couldn’t
handle the myriad of edge conditions that were patently obvious.

The team could see the original problem and asking how to design it
wasn’t adding anything. They knew what to do.

3) Don’t Do: Asking by Providing Their Reason

“Is the reason you don’t click this button because it’s really hard
to see?”

This is a tough one for many observers. We’re watching the
participant and they aren’t doing what we know to be the optimal
method. Our brain races with possible reasons. Need to find out.
Must ask.

“Why didn’t you click this button?” is only a hair better. It
doesn’t telegraph the reasons we’re hypothesizing, but it implies
the person has done something wrong. They’ll tell you a reason, but
it may be more because you’ve pointed it out, not because it’s what
they were actually thinking.

Be creative about exploring the participant’s understanding of the
interface. For example, when I’m curious about why they didn’t click
a particular button, I ask them to talk about what each button does.
That way, they won’t associate the specific button with the action
they just did (and I learn their overall understanding of the
controls).

One Bad Question Won’t End The World

Asking these questions isn’t evil. We’re not violating some
Geneva-Convention-type of international agreement. We won’t be
hauled away by the User Research Police.

However, there is a price to asking the wrong questions. When
conducting user research, the most valuable moments are limited
times that the team spends with each participant. It’s important to
make every second count.

Asking one of these questions wastes those precious moments. They
take time without returning value. The participant tries (and,
usually tries really hard), but that trying doesn’t pay off and eats
down the clock.

Learning to focus on the right questions can help you get the most
out of each session.

Share Your Feedback

Have you compiled your own questions that you shouldn’t ask? Share
your list at the UIE Brain Sparks blog.

About the Author

Jared M. Spool is the founder of User Interface Engineering. He spends his time working with the research teams at the company and helps clients understand how to solve their design problems. You can follow Jared on Twitter at @jmspool.