Menu

Beta — We’re reformatting our articles to work with the new site, but this is one hasn’t been done yet; it may contain formatting issues. No need to report this. We’re working on it.

Using Paper Prototypes to Manage Risk

by Carolyn Snyder
on November 4, 1996

There are no rewards in life for being the first one with the wrong answer. Imagine
spending years building a product, only to learn that it missed the needs of
its intended market. History is littered with the carcasses of failed products
and the companies that built them – product development is indeed a risky business.

We’re a consulting firm specializing in user interface design and usability
issues, so our initial contact with new clients is usually when they realize
their product is in trouble. Here’s a typical scenario: We got a call from
the marketing manager for a commercial software product. The first release
of the product had been around for a year, but sales had been only mediocre.
Although customers liked the functionality, they complained that the software
was difficult to use (the manager admitted that even people within the company
had trouble using it!). The manager was worried that these problems were going
to hurt sales of the next release, and that support costs would escalate. To
manage these risks, the company needed to do something differently.

The company had spent considerable time and money to produce their first release,
just to learn that they hadn’t quite gotten it right. In a sense, that first
release was actually a "prototype" for the next release, a prototype
that had taken many, many developer-years to build. Because their market had
become increasingly competitive, the company couldn’t afford another so-so
release. And they couldn’t wait years or even months for that key feedback
– they needed it before their beta test, which was only a few weeks away.

That’s exactly what we helped them do. Using the techniques of paper prototyping
and usability testing, we showed the team how to get quick feedback on their
product before they had invested a lot of development effort. Total time required:
about 6 days.

Day 1

We met with the entire development team. Like every other team we’ve ever
worked with, these people already had a rich set of opinions (and sometimes,
heated arguments!) about what was wrong with the interface and how to fix it.
The trouble was, they had very little actual data on which problems were most
critical to users or on which of the potential solutions (none of which had
been implemented) were actually going to solve the problems. Without this data,
there was a considerable risk that the team would implement the wrong solutions
or even worse, remain ignorant of where the real problems lay.

The first thing we did with the team was to agree on a profile of their most
important type of user and the most important things that those users did with
the product. There was no time in the schedule to exhaustively test every aspect
of the product, so we had to focus on the riskiest parts – the things that
were going to make or break the next release of the product. In this case,
the team members agreed that the biggest risk was that the product was so hard
to use that it would make a negative impression on the technical buyer. As
another example, in an insurance claims processing system, the biggest risk
might be "Will users accept the way we’ve re-engineered the work flow?" If
the team doesn’t know where the risks lie, that in itself is probably the biggest
risk!

Days 2 and 3

We started looking for people who fit the user profile to bring in for the
usability test sessions. Meanwhile, we showed the team how to construct a paper
prototype of the product interface so we could get some data about those high-risk
areas. Using common office supplies (markers, index cards, scissors, transparency
film) the development team quickly sketched each element of the interface (screen,
menu, error message) on a separate piece of paper.

We completed the prototype by the end of Day 3. It wasn’t very neat – it had
hand-written text, crooked lines and last-minute corrections – but it was good
enough to show what the screens would look like.

Example of a paper prototype

Days 4 and 5

We picked one developer to play the role of "computer," simulating
the behavior of the software by manipulating the pieces of paper. For example,
if the user wanted to open a document in a Windows-based word processor, they
would touch the word "File" on the menu bar. The "computer" would
respond by putting down a piece of paper containing the File menu. The user
would then touch the Open option, the computer would display the Open dialog
box.

We spent 2 days usability testing the paper prototype. In each test session,
we brought in two people who matched the profile of a typical user. We didn’t
give them a demo or explain how to use the interface – instead, we asked them
to do realistic work with the prototype. The computer wasn’t allowed to explain
anything to users, but could only do what the software was designed to do.

We didn’t ask the users for their opinions of the interface; we watched them
actually work with it (we’ve found that sometimes people say they love a product
even though they can’t use it, or vice versa). The entire development team
observed the sessions, taking notes on where the interface was giving the users
problems.

The team was surprised by many of the issues they saw. In some cases, we found
that developers had been arguing about something that the users didn’t even
notice! At the same time, there were huge problems in the interface that no
one had anticipated. Usability testing gave everyone on the team a sense of
what the real issues were; the ones that would affect the success of the next
release.

After each 2-hour test, the team discussed what they had seen and immediately
made changes to the paper prototype. With a paper prototype, the entire team
could collaborate on the design rather than funneling all the input into one
or two programmers. Team members from all disciplines (engineering, marketing,
technical communications, training and management) picked up the markers and
scissors and contributed their expertise directly to the paper prototype.

Modifying a paper prototype is much less painful than for the developers than
modifying an actual product. With a real product, because of the substantial
amount of work they’ve put in, the team has an emotional investment in the
status quo and will naturally tend to "defend" their design. Even
when the team clearly understands the need for changes, it’s tough to throw
away all that hard work. In contrast, because paper prototypes are so easy
to create and modify, there is less invested effort to defend. As a result,
development teams become more flexible and willing to try new ideas. This team
spent about 2 hours making changes.

After lunch, we brought in 2 different users and had them do the same tasks,
to see whether we’d made the product better or worse. Most of the changes proved
to be improvements. Some changes were as simple as using a different word,
moving a button from the top of the screen to the bottom or putting an example
in a help file. Other changes were more substantial – we’ve seen teams completely
redesign their products, literally overnight, after learning that their current
design was headed for disaster.

Day 6

After 2 days of testing, the team had identified and addressed many of the
biggest problems in the interface. On Day 6, we again met with the entire team
to prioritize all the issues that remained in the product, using data – not
opinions – that we had obtained from the usability tests. Given their tight
schedule (they were a month away from beta testing), the team knew where they
had to focus their efforts in order to make the next release a success. On
the seventh day, the team rested.

By using paper prototyping and usability testing, the development team managed
their risks by focusing on them earlier in the project, while there was still
time to make changes. We had started the process by having the team pick those
areas which they had perceived as most critical to the success of the next
release – the areas of highest risk. Now, 6 days later, they had actual data
on how to make the product better in those areas. Our question for the team
was "OK, so now what’s your next greatest risk?" The team now understood
how to use paper prototyping and usability testing to provide the data they
needed for key design decisions that enabled them to build a better product. •