Menu

Beta — We’re reformatting our articles to work with the new site, but this is one hasn’t been done yet; it may contain formatting issues. No need to report this. We’re working on it.

Docs in the Real World

by Carolyn Snyder
on November 1, 1998

In two recent consulting projects, we worked with online documentation developers
who wanted to understand the problems users encountered and how their documentation
helped solve those problems. To find out, we went and observed users in their
own work environments. Although the clients and their software differ significantly,
we found similar issues.

Both development teams initially had specific things they wanted to know about
how people used the documentation, such as whether the structure and organization
made sense to users, and how often users went to the wrong place first.

By watching users in their own workplace, we found issues that we would never
have seen in a usability lab. These observations revealed many larger issues
surrounding the use of the documentation.

We Are Not Alone

For example, neither team expected to see the variety and volume of other
resources that "compete" with the official documentation. These included
such diverse sources as co-workers, including cubicle-mates and the internal
help-desk, users’ own notes, web sites, training materials, online news,
and team meetings.

It surprised both teams to learn how much users relied on these competing
forms of information. They found that the most successful users had the most
resources available and knew when to use each one. No user relied exclusively
on the online documentation.

The teams realized that this competition wasn’t always a problem. For
example, users at one company benefited by having a printed list of field codes
for the mainframe; this information never changed and users often needed it
quickly. (It surprised this team to learn how important the mainframe procedures
were: the company was phasing out the mainframe, so the team hadn’t focused
on supporting it.) In contrast, frequently updated sources such as printed
price schedules quickly became obsolete and many users still had old ones.

Therefore, this team abandoned its original assumption that it had to document
everything in one online source, and decided to look for the most effective
way to convey each type of information. They realized they needed to coordinate
with the training department on how to present the information, including instruction
on using each of the available resources.

Trust-Breakers

We repeatedly observed users avoiding documentation because they’d learned
not to trust it. We saw many reasons for this loss of trust, some of them seemingly
trivial.

Every Error Counts

Even small inaccuracies damaged the confidence of some users. This was more
likely with users who didn’t know how to work around the problem. For
example, users were stopped in their tracks when the online procedures told
them to press Enter — but neglected to tell them they had to press it twice.

Many of these users were not proficient with computers, so even this small
error was enough to make them abandon the documentation and call the help desk.
In contrast, errors in documented policy didn’t affect trust much, because
the support reps usually had more proficiency with the policies — at least
a general idea of what the policy was supposed to be — and could figure
out what to tell the customer.

The Buck Stops Here

Users don’t care where the problem lies: they don’t recognize
that online help involves different pieces from different sources. Although
some problems were beyond the doc team’s control, such as a bookmarks
that didn’t work or a bug in Microsoft’s help engine, they still
affect the users’ perception of the documentation — and their willingness
to use it.

Fixing Broken Trust

It’s easy to break trust, and difficult to restore it. Once they’re
burned by the docs, users typically won’t look there again. Unfortunately,
this behavior can persist even after developers fix a problem. For example,
although one team corrected an error that appeared in Version 2.0 of the docs,
users still didn’t try to use it in Version 4.0. No one had told them
that the error was gone.

When trust takes a beating, it’s hard to know the effect. After a bad
experience, users may be willing to use other parts of the doc — and sometimes
are not. Our observations showed us that the development team needs a concerted,
proactive marketing effort to restore broken trust.

Trust isn’t an all-or-nothing thing. It’s built up slowly by successful
experiences, and quickly eroded by disappointments. One team had translated
the documentation into English from its original German — but unintentionally
left pockets of German. Users who encountered German immediately stopped looking
for the answer (getting German usually meant they were in the wrong place).
Every time they encountered German, users were less likely to rely on the rest
of the documentation.

Trust and Communication

One important element of building trust is the way the documentation team
handles feedback. One team had an internal feed-back line to let users report
errors and request changes. But the team was understaffed, so users rarely
got a response, even an acknowledgment. As a result, users became reluctant
to report problems.

The team realized it needed to put a response system in place — even
if users just got a quick generic thank-you — before it could expect users
to report problems.

Speed Is Relative

Users sometimes avoided documentation if they felt that it took too long to
use. Actual speed, the measurable time it takes to get the answer, often wasn’t
a problem. Instead, perceived lack of speed, how long users think the
process will take, prevented users from using the docs. And what is "acceptable" varies
from user to user, even from situation to situation: with an irate customer
on the line, seconds can seem like hours.

In one project, we saw an interesting Catch-22: support reps didn’t look
up anything in the online documentation unless they already knew where to find
it. But they never learned where the information was unless they spent time
using the docs. This perception of slow access speed wasn’t caused by
poor organization of the documentation, but by the sheer volume of information,
which made the online documentation daunting to the uninitiated.

On the other hand, this team was pleased that users generally could find
things, and that users looked in the wrong place far less often than they’d
feared. So, instead of launching a reorganization, the team proposed some other
changes. These included getting the management to give reps off-phone time
to do research (difficult during busy season), having managers and help-desk
staff encourage reps to look things up first, and devoting more training time
to practice sessions with the online documentation.

Sell, Sell, Sell

The documentation can still fail, even if it’s as usable as possible.
Just because you’ve built it doesn’t mean they will come. We saw
a clear pattern: users generally didn’t know about new features after
they’d been trained.

It amazed both teams when they discovered how many users didn’t know
about some basic features. For example, one team learned that no one had trained
users to press F1 to get help. (Users got very excited when we showed them
this trick!)

The other team found that users never went to the tutorials or used the Back
feature — which the team had spent lots of time developing — because
the team had never told the users about them.

These findings initially discouraged the teams, until they saw it as a "marketing" problem:
determining what the messages they wanted to convey to users about the documentation,
and then finding the appropriate channels.

After seeking feedback from successful reps, one team learned that the most
appropriate and effective marketing message was something like: "The online
documentation is huge and overwhelming at first. But if you invest some effort
in learning it, it will pay off big, making you more proficient in the job
skills you’re rewarded for."

To spread this message, the team decided that managers and the training department
provided the best communication channels.

It Helped to Watch Users

Both teams learned that the documentation was only part of the picture of
how people do their work.

Perfect documents don’t help users succeed unless the team accounts for
all these other factors. Because they made these site visits, the teams came
up with some changes that showed great promise for improving users’ success. •