Menu

Beta — We’re reformatting our articles to work with the new site, but this is one hasn’t been done yet; it may contain formatting issues. No need to report this. We’re working on it.

Watch and Learn: How Recommendation Systems are Redefining the Web

by Joshua Porter
on December 13, 2006

If you’re like most people and enjoy listening to The Beatles, here’s a recommendation:
you might want to check out another British group called Radiohead. Radiohead
has been around for two decades, is widely acclaimed within the music industry,
and has a very devout following, especially in Britain.

But you may not have heard of Radiohead. Their 1993 alternative rock smash-hit, "Creep",
arguably their best-known song, is as melodramatic as they come and very un-Beatles-like.
Moreover, Radiohead doesn’t play the kind of pop music that made the Beatles
famous. Although Radiohead explores a wide variety of sounds, their music can
best be described as alternative-rock or punk-rock. Whereas the Beatles took
the world by storm, Radiohead took their fame by stealth.

Why then, would a Beatles fan like Radiohead? Well, the proof is in the pudding.
The Last.fm pudding, that is. Last.fm is a social music site that aggregates
what music its members listen to and tallies the results. It looks to see what
artists are played alongside other artists. For example, for all those people
who listen to the Beatles, what other artists do they listen to?

The number one match is Bob Dylan, which makes sense since many of the
people who grew up with the Beatles also grew up with Dylan. We might expect
those two together. However, number two on the list is Radiohead, from a completely
different generation and genre of music. According to Last.fm, Beatles fans
listen to a lot of Radiohead, and vice versa.

Last.fm Beatles

Recommendation Systems

Last.fm is one of a relatively new breed of web applications
called recommendation systems. Recommendation systems aggregate the online
behavior of many people to find trends and make recommendations based on them.
This involves mathematically calculating how one person’s actions relate to
another’s. But we can think of it as simply finding people who act similarly
when listening to music, rating movies, or reading news articles. And the results
are sometimes unexpected. For example, who knew Beatles fans might enjoy listening
to Radiohead?

In addition to Last.fm, the iTunes
music store
from Apple has a "Just
for You" feature that recommends songs and albums. Amazon.com has
long had a recommendations feature, recommending everything from kitchen tools
to computers to books. Netflix.com recommends
movies and TV shows to watch. Yahoo,
IMDB, Rotten
Tomatoes
, and Metacritic all recommend
movies too. These are all straight-forward recommendation systems. In fact,
some even tell you so.

Other applications, such as Digg.com, also draw on the same idea of aggregating
the actions of many people to present content. Digg and Reddit harness users’ actions
to recommend top news stories. The New York Times has a "Most
Popular
" section
that displays the most emailed and most blogged stories. Del.icio.us has a "popular" page
that displays the most-saved bookmarks. TripAdvisor recommends lodging and
places to visit. Techmeme, a social news site, counts links on blogs as votes
to recommend news from the tech industry. Flickr has a photo recommendation
feature they call "interestingness." It’s also not too much
of a stretch to think of Google as a recommendation system, recommending results
based on the search keywords users enter.

The Benefits of Recommendation Systems

The success of applications that recommend
is growing. Recommendation systems are no longer a novelty. They’re being built
for almost every domain where we can give recommendations, and their advantages
are clear.

  • Based on Real Activity
    The biggest benefit of recommendation systems is
    that they record, and then base their recommendations on actual user behavior.
    Their recommendations are not based on guesswork, but on an objective reality.
    This is the holy grail of design: watching people in their natural environment
    and making design decisions based directly on the results. Recommendation
    systems are not perfect, but because they predict the future based on the
    past, they are remarkably good.
  • Great for Discovery
    Sometimes it can feel like we’re the victims of horribly
    inefficient advertising. This is apparent when we go to the movies and none
    of the previews are interesting, or when none of the music on the radio is
    aligned to our tastes. Recommendation systems help alleviate this problem
    because they allow us to discover things that are similar to what we already
    like. And, as in the Beatles/Radiohead example, they can make some pretty
    surprising recommendations that we probably wouldn’t have found out about
    otherwise.
  • Personalization
    We often take recommendations from friends and family because
    we trust their opinion. Part of that trust is built on our relationship:
    they know us better than anyone else so they know what we like and what we
    don’t like. They’re good at recommending things for that very reason. This
    is what recommendation systems try to model: the intimate knowledge of what
    we like and don’t like.
  • Always Up-To-Date
    Recommendation systems are dynamically updated, and therefore
    are always up-to-date. Some new, interesting news? It will be on Digg within
    minutes. An interesting new product on Amazon? It quickly gets recommended
    as long as people rate it highly. The ability for a recommendation system
    to bubble up activity in real time is a huge advantage because the system
    is always on.
  • Reduced Organizational Maintenance
    Building recommendation systems is quite
    different from how we’ve built information-rich web sites in the past. For
    many designers the primary task of building an information-rich web site
    is creating navigation systems built on top of an underlying taxonomy. The
    taxonomy is built out of the designer’s knowledge of users and the domain,
    generated from observations made during field research, insights from persona
    creation, or knowledge gained from other design techniques. Most of the organizational
    maintenance of a site is keeping the navigation system and taxonomy in line
    with the users’ changing needs.

    With recommendation systems, much of
    this organizational maintenance goes away. The users organize their own content,
    in a sense, as the system monitors their constant activity to decide what navigation
    options go where. What related links could go on a page dedicated to the Beatles?
    Bob Dylan and Radiohead, for starters. This doesn’t mean that navigation-related
    decisions go away, however. It still takes a designer to decide what type of
    information should be displayed on what screen.

Drawbacks

Recommendation systems are not perfect. They have several drawbacks
that design teams should consider if they’re thinking about implementing
them.

  • Difficult to Set Up
    Recommendation systems are intensive, database-driven
    applications that are not trivial to create and get running. It takes a serious
    development project to do so. Moving to a recommendation system takes time,
    energy, and a long-term commitment.
  • Maintenance Shifted Elsewhere
    Even though recommendation systems can reduce
    organizational maintenance, they don’t get rid of maintenance itself. For
    example, keeping the system up and running becomes a major task, as it is
    with any significant database-backed system.
  • Sometimes They’re Wrong
    Recommendation systems aren’t just a technological
    challenge. They’re also a social one. Sometimes people are unhappy with recommendations.
    Sometimes recommendations are wrong. In extreme cases recommendations can
    be offensive. For example, Wal-Mart
    got into serious hot water
    last year when its movie
    recommendation system recommended movies in an inappropriate way (it still
    hasn’t put its recommendation system back online.) Similarly, Amazon
    took some heat
    when it started cross-promoting its new clothing site
    by recommending clean underwear to people who were shopping for DVDs.
  • Gaming
    Gaming is a big issue, especially on news recommendation sites. Digg,
    the most popular site of this kind, is continually battling attempts by users
    to game the system. When a news story is popular on Digg, it gets promoted
    to the home page as a recommendation for everyone to click on and read. When
    this happens the clicking and reading can quickly get into the thousands.
    (Many web sites have actually been brought down by the resulting traffic.)
    This huge increase in attention is attractive to those people willing to
    game the system for their own personal benefit. Digg is constantly moderating
    its members to keep this at a minimum.

Going Forward

Recommendation systems are popping up everywhere, from movies
to news to travel and leisure. They provide valuable, personalized information
that can greatly influence the way we use the Web. Like any system, however,
they are not without their faults.  But their benefits seem to outweigh
their drawbacks, and they might even be the beginnings of an artificial intelligence
for each of us, letting us find our next Radiohead. Recommendation systems
are out there: watching, and learning. •