Menu

Beta — We’re reformatting our articles to work with the new site, but this is one hasn’t been done yet; it may contain formatting issues. No need to report this. We’re working on it.

Six Slick Tests for Docs and Help

by Tara Scanlon
on September 1, 1998

Usability testing isn’t just for software and web sites. Testing documentation
can ensure that it includes — and accurately conveys — all the information
users expect and need.

Testing gives you accurate information on how well your documentation and
Help work. It can even uncover problems that are better solved by changing
the interface.

As with most parts of the development, it’s easier and less expensive
to find and correct doc problems early in the process, so we try to test as
soon as possible.

Choosing a method depends on what you want to learn, how much time you have,
and where your greatest risks are.

1. Incredibly Intelligent Help

This technique lets you discover what information users actually need, before
you’ve written anything.

We bring in test participants as soon as we have a version of the product
that will let users do some of the tasks they’d do in the real world.
This may be well before we’ve written the documentation, but this isn’t
a problem because a member of the development team simulates the documentation.

The incredibly intelligent part comes in when users get stuck or seem confused.
At this point, the person simulating the docs asks, "What is your question
right now?" When the user replies, the Help person provides a verbal clue,
but gives as small a piece of information as possible. If this isn’t enough,
the user can ask another question.

By giving information only a bit at a time, we can find out exactly which
piece is most important to the user. Knowing this helps teams provide appropriate
documentation — not too much, not too little — or, better yet, to
fix the interface itself.

2. Index Tests

You can determine whether users can get to the right topic in your index
by having them work with the draft index. You can even test the index before
you’ve written the documentation.

To test the index, we show users screen shots from the product and ask them
to write down the first three terms they’d look up to find more information.
This test lets you look inside users’ heads and learn some of the terms
they use — and expect to see. We write down all the terms they use; they
could be words we’ve omitted or not referenced. The more of these we include,
the better the index will work.

3. Summary Tests

By testing how well users’ comprehend conceptual material, you’ll
know how successfully you’re communicating the information.

We ask users to read a section or examine a picture in the docs, then ask
them to describe the three main points this element is trying to convey. (You
should already know what you expect them to be!) If the users’ answers
match yours, you’re probably getting your information across; if not,
you may have some work to do.

4. Procedural Tests

This is a good way to test how accurately you’ve written the procedural
details of your docs.

We give users the documentation or Help, asking them to read a specific page
or topic and then work with the software to complete the procedure this section
describes.

We note where they have problems, where they get confused, or where steps
are missing. Then we make the necessary changes in the next draft.

5. Lookup Tests

Lookup tests demonstrate if the docs or Help include the necessary information — and
if users can get to the right topic and use it. They also let you find the
users’ biggest unanswered question.

To do a lookup test, we give users questions to answer or tasks to complete
and see if the draft documentation helps them succeed. For example, when testing
the documentation for a video product, we told users, "The picture you’re
getting isn’t as clear as you’d like. Use the documentation and online
help to find out what’s wrong and how to fix it."

After conducting lookup tests, some teams have changed from printed to online
help (or vice versa), added index entries, or reorganized the information.

6. Rose in the Thorns Tests

It can be enlightening — and fascinating — to discover what experienced
users don’t know about a product. To do this, we’ve recently
started experimenting with the Rose in the Thorns test.

We ask experienced users to look through the documentation or Help until they
find information about features they didn’t know before.

Then we ask them if this new information is valuable to them. If there are
important features users don’t know about, we work with the interface
designers to communicate them better.

Tips for Success

Three things help ensure successful tests of docs and help:

  • These tests work best with a couple of users at a time. This increases
    the likelihood of brainstorming and lets us collect lots of information or
    keywords at once.
  • We try to avoid product-specific terms in the task instructions.
  • After the tests, we discuss with the team what we’ve seen and decide
    on next steps — including changes to the docs or more testing.