Why On-Site Searching Stinks
In our most recent web-site studies, we watched users look for information within
web sites. Our goal was to gather data about what makes a good link, but we
did not tell users whether or not they should use the site’s search facilities.
Users went to these search engines in almost half the tasks. Maybe they shouldn’t
Searching Reduces Success
Using an on-site search engine actually reduced the chances of success,
and the difference was significant. Overall, users found the correct answer
in 42% of the tests. When they used an on-site search engine (we did not study
Internet search engines), their success rate was only 30%. In tasks where they
used only links, however, users succeeded 53% of the time.
Garbage In/Garbage Out
A search engine’s results can only be as good as the inputs it receives. Our
tests showed that users didn’t know what to type in a search edit box, or they
didn’t know how to use the format the search engine expected.
Users didn’t seem to understand that some search engines distinguish between
partial and entire words. On the Car Talk site (cartalk.msn.com) they didn’t
realize that a search for “tire” would give them results that contained the
word entire or the phrase I’m tired.
Although the site gives users the option of searching for whole or partial
words, no one changed the setting. Similarly, we saw that users were puzzled
when plural and singular words produced different results. Prefix/suffix strippers might
improve the quality of the search inputs.
Few users in our studies clicked to bring up search tips that were located
on another page. We believe this shows that few users read search instructions,
especially instructions that appear below the keyword box.
Lots of users misspell keywords. An analysis of a week’s worth of logs from
Netscape’s DevEdge Online site showed that 3% of searches contained misspelled
cases, the user was only off by a single letter:
Reasons for spelling errors include unfamiliarity with the English language
or technical terms, difficulty in spelling, or simply hitting the wrong key.
Users didn’t always know that typing errors would produce poor results and
couldn’t tell that the “no matches found” message was caused by a typo.
A spell-checker would weed out some problems, especially if
it suggested alternate spellings (Microsoft Word’s spell-checker is smart enough
to suggest “videos” as a possibility when you type “vidoes.”)
Problems Interpreting Search Results
Correctly asking for the right information is only half the battle. Many users
had problems even after they got the search results.
Users often had trouble determining why a search
returned a particular item. Many search results offer no clue and can look
like the example here from a search for “money market” on the Fidelity site
(www.fidelity.com). Car Talk does better because it shows keywords in the context
of a line, so users could weed out obviously irrelevant hits in the search
Some on-site engines generate links by looking at page title such as the result
6 from Fidelity, above, “funddata_toc.html.” In this case, the designer apparently
did not use the HTML <title> tag. When designers create page titles,
they may not be thinking about how those titles will look when they turn up
in search results. Taking the time to create more descriptive page titles might
help. This is one of the few search engine problems designers can control.
The Perils of Full-Text Searches
Most sites use a full-text search, which sometimes returned irrelevant and
often amusing results. In a keyword search for “dinosaur” on the Smithsonian
Magazine site, the first result led to an article about the steel industry!
We saw many other instances of this problem. Searching for “wild west” on
the Discovery site returned a link to an article on sharks — though the
article did indeed contain both those words.
A full-text search is a blunt instrument for chipping away at a large block
of information in order to sculpt the desired result. An index is a more precise
tool. No self-respecting human indexer would have referenced the steel industry
article under “dinosaur.” Good indexing is a skill; humans do it better than
machines. We anticipate that professional indexers may become more involved
in web site design in the future. This may be difficult and expensive to implement,
but the resulting user satisfaction might make it worthwhile.
Worse than Nothing
Our data showed that today’s on-site search engines are worse than nothing — significantly
worse. Searching is a difficult problem with no solution visible on the horizon.
Until the technology is equal to the challenge, we suggest that designers seriously
consider not including a search engine on their sites until the technology
is equal to the challenge.
In the meantime, our testing data suggests that designers would have more
success by focusing instead on creating effective links. •