Watch and Learn: How Recommendation Systems are Redefining the Web
If you’re like most people and enjoy listening to The Beatles, here’s a recommendation:
you might want to check out another British group called Radiohead. Radiohead
has been around for two decades, is widely acclaimed within the music industry,
and has a very devout following, especially in Britain.
But you may not have heard of Radiohead. Their 1993 alternative rock smash-hit, "Creep",
arguably their best-known song, is as melodramatic as they come and very un-Beatles-like.
Moreover, Radiohead doesn’t play the kind of pop music that made the Beatles
famous. Although Radiohead explores a wide variety of sounds, their music can
best be described as alternative-rock or punk-rock. Whereas the Beatles took
the world by storm, Radiohead took their fame by stealth.
Why then, would a Beatles fan like Radiohead? Well, the proof is in the pudding.
The Last.fm pudding, that is. Last.fm is a social music site that aggregates
what music its members listen to and tallies the results. It looks to see what
artists are played alongside other artists. For example, for all those people
who listen to the Beatles, what other artists do they listen to?
The number one match is Bob Dylan, which makes sense since many of the
people who grew up with the Beatles also grew up with Dylan. We might expect
those two together. However, number two on the list is Radiohead, from a completely
different generation and genre of music. According to Last.fm, Beatles fans
listen to a lot of Radiohead, and vice versa.
Last.fm is one of a relatively new breed of web applications
called recommendation systems. Recommendation systems aggregate the online
behavior of many people to find trends and make recommendations based on them.
This involves mathematically calculating how one person’s actions relate to
another’s. But we can think of it as simply finding people who act similarly
when listening to music, rating movies, or reading news articles. And the results
are sometimes unexpected. For example, who knew Beatles fans might enjoy listening
In addition to Last.fm, the iTunes
music store from Apple has a "Just
for You" feature that recommends songs and albums. Amazon.com has
long had a recommendations feature, recommending everything from kitchen tools
to computers to books. Netflix.com recommends
movies and TV shows to watch. Yahoo,
Tomatoes, and Metacritic all recommend
movies too. These are all straight-forward recommendation systems. In fact,
some even tell you so.
Other applications, such as Digg.com, also draw on the same idea of aggregating
the actions of many people to present content. Digg and Reddit harness users’ actions
to recommend top news stories. The New York Times has a "Most
that displays the most emailed and most blogged stories. Del.icio.us has a "popular" page
that displays the most-saved bookmarks. TripAdvisor recommends lodging and
places to visit. Techmeme, a social news site, counts links on blogs as votes
to recommend news from the tech industry. Flickr has a photo recommendation
feature they call "interestingness." It’s also not too much
of a stretch to think of Google as a recommendation system, recommending results
based on the search keywords users enter.
The Benefits of Recommendation Systems
The success of applications that recommend
is growing. Recommendation systems are no longer a novelty. They’re being built
for almost every domain where we can give recommendations, and their advantages
- Based on Real Activity
The biggest benefit of recommendation systems is
that they record, and then base their recommendations on actual user behavior.
Their recommendations are not based on guesswork, but on an objective reality.
This is the holy grail of design: watching people in their natural environment
and making design decisions based directly on the results. Recommendation
systems are not perfect, but because they predict the future based on the
past, they are remarkably good.
- Great for Discovery
Sometimes it can feel like we’re the victims of horribly
inefficient advertising. This is apparent when we go to the movies and none
of the previews are interesting, or when none of the music on the radio is
aligned to our tastes. Recommendation systems help alleviate this problem
because they allow us to discover things that are similar to what we already
like. And, as in the Beatles/Radiohead example, they can make some pretty
surprising recommendations that we probably wouldn’t have found out about
We often take recommendations from friends and family because
we trust their opinion. Part of that trust is built on our relationship:
they know us better than anyone else so they know what we like and what we
don’t like. They’re good at recommending things for that very reason. This
is what recommendation systems try to model: the intimate knowledge of what
we like and don’t like.
- Always Up-To-Date
Recommendation systems are dynamically updated, and therefore
are always up-to-date. Some new, interesting news? It will be on Digg within
minutes. An interesting new product on Amazon? It quickly gets recommended
as long as people rate it highly. The ability for a recommendation system
to bubble up activity in real time is a huge advantage because the system
is always on.
- Reduced Organizational Maintenance
Building recommendation systems is quite
different from how we’ve built information-rich web sites in the past. For
many designers the primary task of building an information-rich web site
is creating navigation systems built on top of an underlying taxonomy. The
taxonomy is built out of the designer’s knowledge of users and the domain,
generated from observations made during field research, insights from persona
creation, or knowledge gained from other design techniques. Most of the organizational
maintenance of a site is keeping the navigation system and taxonomy in line
with the users’ changing needs.
With recommendation systems, much of
this organizational maintenance goes away. The users organize their own content,
in a sense, as the system monitors their constant activity to decide what navigation
options go where. What related links could go on a page dedicated to the Beatles?
Bob Dylan and Radiohead, for starters. This doesn’t mean that navigation-related
decisions go away, however. It still takes a designer to decide what type of
information should be displayed on what screen.
Recommendation systems are not perfect. They have several drawbacks
that design teams should consider if they’re thinking about implementing
- Difficult to Set Up
Recommendation systems are intensive, database-driven
applications that are not trivial to create and get running. It takes a serious
development project to do so. Moving to a recommendation system takes time,
energy, and a long-term commitment.
- Maintenance Shifted Elsewhere
Even though recommendation systems can reduce
organizational maintenance, they don’t get rid of maintenance itself. For
example, keeping the system up and running becomes a major task, as it is
with any significant database-backed system.
- Sometimes They’re Wrong
Recommendation systems aren’t just a technological
challenge. They’re also a social one. Sometimes people are unhappy with recommendations.
Sometimes recommendations are wrong. In extreme cases recommendations can
be offensive. For example, Wal-Mart
got into serious hot water last year when its movie
recommendation system recommended movies in an inappropriate way (it still
hasn’t put its recommendation system back online.) Similarly, Amazon
took some heat when it started cross-promoting its new clothing site
by recommending clean underwear to people who were shopping for DVDs.
Gaming is a big issue, especially on news recommendation sites. Digg,
the most popular site of this kind, is continually battling attempts by users
to game the system. When a news story is popular on Digg, it gets promoted
to the home page as a recommendation for everyone to click on and read. When
this happens the clicking and reading can quickly get into the thousands.
(Many web sites have actually been brought down by the resulting traffic.)
This huge increase in attention is attractive to those people willing to
game the system for their own personal benefit. Digg is constantly moderating
its members to keep this at a minimum.
Recommendation systems are popping up everywhere, from movies
to news to travel and leisure. They provide valuable, personalized information
that can greatly influence the way we use the Web. Like any system, however,
they are not without their faults. But their benefits seem to outweigh
their drawbacks, and they might even be the beginnings of an artificial intelligence
for each of us, letting us find our next Radiohead. Recommendation systems
are out there: watching, and learning. •