Menu

Beta — We’re reformatting our articles to work with the new site, but this is one hasn’t been done yet; it may contain formatting issues. No need to report this. We’re working on it.

Moderating with Multiple Personalities: 3 Roles for Facilitating Usability Tests

by Jared M. Spool
on October 14, 2009

Editor’s note: This article is avialable in German.

The team was psyched. On one level, it was hard to imagine why.
After all, the last two days was filled with a pile of bad news:
many parts of the design need rethinking.

Surprisingly, with all the bad news, the team wasn’t depressed.
Eight, shiny usability test participants just read them the riot
act; demonstrating a slew of places where the design needs
rethinking. Yet the team was energized and ready to tackle the
challenge.

Usability tests are a core design tool and, when done well, they
deliver tremendous insights to the team. However, when a usability
test is done poorly, it can be a disaster for everyone involved. An
important key to their success is the work of a great moderator.

Great Moderators Are Grown, Not Bought

The best usability test moderators have a lot in common with an
orchestra conductor. They keep the participant comfortable and
stress-free. The moderator tries to make the participant forget they
are in a foreign environment with a bunch of strangers who intensely
watch everything that he/she does. They keep the information flowing
to the design team, especially the tough news. And they do all this
with organized flair and patience, ensuring every aspect of the
user’s experience is explored.

How do you become a great moderator? Simple—with practice.
Facilitating sessions is a learned skill that improves the more you
do it. There are some simple tricks and techniques behind it. Once
you learn those, and have a chance to practice them, you too can
become a top-notch moderator.

An important trick to moderating is mastering the multiple
personalities involved. A usability test moderator serves three
parties: the participant, the session observers, and the rest of the
design team. To make this happen, a good facilitator adopts three
personalities simultaneously: the flight attendant, the
sportscaster, and the scientist.

Priority #1: The Flight Attendant Personality

Being a participant in a usability test can be quite stressful. You
arrive at a strange building. There are people watching you, or
worse, there’s a big glass mirror with who-knows-what crazy things
behind it. Moreover, people keep calling it a "test", so you have to
be on your best performance.

When the moderator adopts the flight attendant personality, they’re
watching out for the participant’s comfort and safety. From the
moment the participant walks in the door, the moderator helps them
feel at home. They get them coffee, explain the procedure, and
answer questions. (The best moderators start before the participant
arrives, by working with the recruiters to set the right
expectations and answer any questions.)

During the session, they smile a lot, keeping the session relaxed.
They watch diligently for any signs of stress. When the interface
isn’t working well or the participant is struggling, they give
reassuring messages: "You’re doing well." "This is helping a lot."
"You’re helping us discover problems we didn’t realize we had."

It’s the flight attendant’s responsibility to stop the session if
the participant becomes too stressed. The flight attendant also
knows when to encourage the participant to keep trying, when the
interface becomes a little challenging. And the flight attendant
helps the participant through the bumpy bits, so they can move on to
working with the rest of the design.

Safety and comfort: that’s the flight attendant’s focus.

Priority #2: The Sportscaster Personality

The sportscaster personality’s job is to make sure every observer in
the session catches all of the action.

When we’re facilitating usability tests, we start by setting up a
projector in the room, so it’s easy for the observers to see what’s
on the participant’s screen. We encourage the participant to "think
out loud", letting us know what’s going through their head as they
use the design.

For those participants that are naturally quiet, we engage in a
"color commentary", where we repeat and narrate the activity. (While
you think this might be unnerving for the participants, we’ve found
they actually get into the rhythm quite quickly. It’s common for a
participant to fill in the narration at moments when the moderator
becomes distracted with something else.)

The sportscaster looks for the exciting portions of the session and
makes them last. When a participant runs into a tricky part of the
design, the sportscaster kicks in to ask questions to better
understand the participant’s viewpoint. Do they know there’s a
problem? Do they understand what the design is trying to tell them?
Do they have a strategy for resolving the issue? What terms do they
use to explain what’s happening?

The sportscaster knows her audience. She caters the session to the
folks who are watching. Recently, while moderating a test of an
e-commerce web site, I guided the participant into an area that we
knew had problems, because I knew the senior manager in charge was
observing their first session. This helped the designers get support
for making some difficult decisions.

Catching all the action: that’s the sportscaster’s focus.

Priority #3: The Scientist Personality

The scientist personality looks for the data. Since the goal of any
user research is to help the team make better design decisions, the
scientist is there to collect the data and help the team analyze it.

Like the other roles, this starts long before the participant shows
up. The scientist puts together the test plans, deciding the tasks
the participants will try. The scientist creates questionnaires and
interview scripts, to learn more about the participant’s background
and experience. Every thing the scientist does is to make sure the
team collects every piece of data they’ll need.

Part of the preparation involves how the findings are used once the
sessions are completed. How will the team analyze and synthesize
this information? We like to use a technique we learned from
seasoned usability testing manager, Mary Beth Rettger. Prior to each
session, large yellow sticky notes are passed to each observer.
Then, they are asked to jot down their observations, one per note.
After the last test session, we do a quick KJ analysis
to determine the groupings of what we saw
and their priorities.

Guiding the data collection: that’s the scientist’s focus.

Watching the Priorities

The order of the three priorities is critical. A great moderator
first watches out for the participants, putting them at ease and
ensuring their comfort.

The sportscaster is the second priority to the flight attendant.
While we want to explore everything that’s happening in the session,
we have to be cognizant of the participant’s safety and comfort—
that always comes first.

The scientist is the last of the priorities. We have to know that,
if we’d done our job correctly, important data won’t be lost, even
if we had to focus on the participant and the observers to get it.

For this reason, we recommend that moderators don’t take notes until
they’ve really practiced the flight attendant and sportscaster
roles. Instead, they can delegate the note taking to an observer,
who can focus on collecting everything. Only after they’ve mastered
those two roles should they attempt to take notes during the
session.

Once you master the three priorities, you’ll find it easy to get the
team excited about testing. They’ll come out of the session
energized and itching to make improvements. And that’s what good
user research is all about.

Share Your Thoughts with Us

Have you tried moderating usability tests? Post your thoughts at questions at our Brain Sparks Blog.

About the Author

Jared M. Spool is the founder of User Interface Engineering. He spends his time working with the research teams at the company and helps clients understand how to solve their design problems. You can follow Jared on Twitter at @jmspool.