Moderating with Multiple Personalities: 3 Roles for Facilitating Usability Tests

by Jared M. Spool

Thanks to Seibert Media for the German translation to this article.

The team was psyched. On one level, it was hard to imagine why. After all, the last two days was filled with a pile of bad news: many parts of the design need rethinking.

Surprisingly, with all the bad news, the team wasn’t depressed. Eight, shiny usability test participants just read them the riot act; demonstrating a slew of places where the design needs rethinking. Yet the team was energized and ready to tackle the challenge.

Usability tests are a core design tool and, when done well, they deliver tremendous insights to the team. However, when a usability test is done poorly, it can be a disaster for everyone involved. An important key to their success is the work of a great moderator.

Great Moderators Are Grown, Not Bought

The best usability test moderators have a lot in common with an orchestra conductor. They keep the participant comfortable and stress-free. The moderator tries to make the participant forget they are in a foreign environment with a bunch of strangers who intensely watch everything that he/she does. They keep the information flowing to the design team, especially the tough news. And they do all this with organized flair and patience, ensuring every aspect of the user’s experience is explored.

How do you become a great moderator? Simple—with practice. Facilitating sessions is a learned skill that improves the more you do it. There are some simple tricks and techniques behind it. Once you learn those, and have a chance to practice them, you too can become a top-notch moderator.

An important trick to moderating is mastering the multiple personalities involved. A usability test moderator serves three parties: the participant, the session observers, and the rest of the design team. To make this happen, a good facilitator adopts three personalities simultaneously: the flight attendant, the sportscaster, and the scientist.

Priority #1: The Flight Attendant Personality

Being a participant in a usability test can be quite stressful. You arrive at a strange building. There are people watching you, or worse, there’s a big glass mirror with who-knows-what crazy things behind it. Moreover, people keep calling it a “test”, so you have to be on your best performance.

When the moderator adopts the flight attendant personality, they’re watching out for the participant’s comfort and safety. From the moment the participant walks in the door, the moderator helps them feel at home. They get them coffee, explain the procedure, and answer questions. (The best moderators start before the participant arrives, by working with the recruiters to set the right expectations and answer any questions.)

During the session, they smile a lot, keeping the session relaxed. They watch diligently for any signs of stress. When the interface isn’t working well or the participant is struggling, they give reassuring messages: “You’re doing well.” “This is helping a lot.” “You’re helping us discover problems we didn’t realize we had.”

It’s the flight attendant’s responsibility to stop the session if the participant becomes too stressed. The flight attendant also knows when to encourage the participant to keep trying, when the interface becomes a little challenging. And the flight attendant helps the participant through the bumpy bits, so they can move on to working with the rest of the design.

Safety and comfort: that’s the flight attendant’s focus.

Priority #2: The Sportscaster Personality

The sportscaster personality’s job is to make sure every observer in the session catches all of the action.

When we’re facilitating usability tests, we start by setting up a projector in the room, so it’s easy for the observers to see what’s on the participant’s screen. We encourage the participant to “think out loud”, letting us know what’s going through their head as they use the design.

For those participants that are naturally quiet, we engage in a “color commentary”, where we repeat and narrate the activity. (While you think this might be unnerving for the participants, we’ve found they actually get into the rhythm quite quickly. It’s common for a participant to fill in the narration at moments when the moderator becomes distracted with something else.)

The sportscaster looks for the exciting portions of the session and makes them last. When a participant runs into a tricky part of the design, the sportscaster kicks in to ask questions to better understand the participant’s viewpoint. Do they know there’s a problem? Do they understand what the design is trying to tell them? Do they have a strategy for resolving the issue? What terms do they use to explain what’s happening?

The sportscaster knows her audience. She caters the session to the folks who are watching. Recently, while moderating a test of an e-commerce web site, I guided the participant into an area that we knew had problems, because I knew the senior manager in charge was observing their first session. This helped the designers get support for making some difficult decisions.

Catching all the action: that’s the sportscaster’s focus.

Priority #3: The Scientist Personality

The scientist personality looks for the data. Since the goal of any user research is to help the team make better design decisions, the scientist is there to collect the data and help the team analyze it.

Like the other roles, this starts long before the participant shows up. The scientist puts together the test plans, deciding the tasks the participants will try. The scientist creates questionnaires and interview scripts, to learn more about the participant’s background and experience. Every thing the scientist does is to make sure the team collects every piece of data they’ll need.

Part of the preparation involves how the findings are used once the sessions are completed. How will the team analyze and synthesize this information? We like to use a technique we learned from seasoned usability testing manager, Mary Beth Rettger. Prior to each session, large yellow sticky notes are passed to each observer. Then, they are asked to jot down their observations, one per note. After the last test session, we do a quick KJ analysis to determine the groupings of what we saw and their priorities.

Guiding the data collection: that’s the scientist’s focus.

Watching the Priorities

The order of the three priorities is critical. A great moderator first watches out for the participants, putting them at ease and ensuring their comfort.

The sportscaster is the second priority to the flight attendant. While we want to explore everything that’s happening in the session, we have to be cognizant of the participant’s safety and comfort—that always comes first.

The scientist is the last of the priorities. We have to know that, if we’d done our job correctly, important data won’t be lost, even if we had to focus on the participant and the observers to get it.

For this reason, we recommend that moderators don’t take notes until they’ve really practiced the flight attendant and sportscaster roles. Instead, they can delegate the note taking to an observer, who can focus on collecting everything. Only after they’ve mastered those two roles should they attempt to take notes during the session.

Once you master the three priorities, you’ll find it easy to get the team excited about testing. They’ll come out of the session energized and itching to make improvements. And that’s what good user research is all about.

Published here on October 14, 2009.