Watch and Learn: How Recommendation Systems are Redefining the Web

If you’re like most people and enjoy listening to The Beatles, here’s a recommendation: you might want to check out another British group called Radiohead. Radiohead has been around for two decades, is widely acclaimed within the music industry, and has a very devout following, especially in Britain.

But you may not have heard of Radiohead. Their 1993 alternative rock smash-hit, “Creep”, arguably their best-known song, is as melodramatic as they come and very un-Beatles-like. Moreover, Radiohead doesn’t play the kind of pop music that made the Beatles famous. Although Radiohead explores a wide variety of sounds, their music can best be described as alternative-rock or punk-rock. Whereas the Beatles took the world by storm, Radiohead took their fame by stealth.

Why then, would a Beatles fan like Radiohead? Well, the proof is in the pudding. The Last.fm pudding, that is. Last.fm is a social music site that aggregates what music its members listen to and tallies the results. It looks to see what artists are played alongside other artists. For example, for all those people who listen to the Beatles, what other artists do they listen to?

The number one match is Bob Dylan, which makes sense since many of the people who grew up with the Beatles also grew up with Dylan. We might expect those two together. However, number two on the list is Radiohead, from a completely different generation and genre of music. According to Last.fm, Beatles fans listen to a lot of Radiohead, and vice versa.

Recommendation Systems

Last.fm is one of a relatively new breed of web applications called recommendation systems. Recommendation systems aggregate the online behavior of many people to find trends and make recommendations based on them. This involves mathematically calculating how one person’s actions relate to another’s. But we can think of it as simply finding people who act similarly when listening to music, rating movies, or reading news articles. And the results are sometimes unexpected. For example, who knew Beatles fans might enjoy listening to Radiohead?

In addition to Last.fm, the iTunes music store from Apple has a “Just for You” feature that recommends songs and albums. Amazon.com has long had a recommendations feature, recommending everything from kitchen tools to computers to books. Netflix.com recommends movies and TV shows to watch. Yahoo, IMDB, Rotten Tomatoes, and Metacritic all recommend movies too. These are all straight-forward recommendation systems. In fact, some even tell you so.

Other applications, such as Digg.com, also draw on the same idea of aggregating the actions of many people to present content. Digg and Reddit harness users’ actions to recommend top news stories. The New York Times has a “Most Popular” section that displays the most emailed and most blogged stories. Del.icio.us has a “popular” page that displays the most-saved bookmarks. TripAdvisor recommends lodging and places to visit. Techmeme, a social news site, counts links on blogs as votes to recommend news from the tech industry. Flickr has a photo recommendation feature they call “interestingness.” It’s also not too much of a stretch to think of Google as a recommendation system, recommending results based on the search keywords users enter.

The Benefits of Recommendation Systems

The success of applications that recommend is growing. Recommendation systems are no longer a novelty. They’re being built for almost every domain where we can give recommendations, and their advantages are clear.

• Based on Real Activity

  The biggest benefit of recommendation systems is that they record, and then base their recommendations on actual user behavior. Their recommendations are not based on guesswork, but on an objective reality. This is the holy grail of design: watching people in their natural environment and making design decisions based directly on the results. Recommendation systems are not perfect, but because they predict the future based on the past, they are remarkably good.

• Great for Discovery

  Sometimes it can feel like we’re the victims of horribly inefficient advertising. This is apparent when we go to the movies and none of the previews are interesting, or when none of the music on the radio is aligned to our tastes. Recommendation systems help alleviate this problem because they allow us to discover things that are similar to what we already like. And, as in the Beatles/Radiohead example, they can make some pretty surprising recommendations that we probably wouldn’t have found out about otherwise.

• Personalization

  We often take recommendations from friends and family because we trust their opinion. Part of that trust is built on our relationship: they know us better than anyone else so they know what we like and what we don’t like. They’re good at recommending things for that very reason. This is what recommendation systems try to model: the intimate knowledge of what we like and don’t like.

• Always Up-To-Date

  Recommendation systems are dynamically updated, and therefore are always up-to-date. Some new, interesting news? It will be on Digg within minutes. An interesting new product on Amazon? It quickly gets recommended as long as people rate it highly. The ability for a recommendation system to bubble up activity in real time is a huge advantage because the system is always on.

• Reduced Organizational Maintenance

  Building recommendation systems is quite different from how we’ve built information-rich web sites in the past. For many designers the primary task of building an information-rich web site is creating navigation systems built on top of an underlying taxonomy. The taxonomy is built out of the designer’s knowledge of users and the domain, generated from observations made during field research, insights from persona creation, or knowledge gained from other design techniques. Most of the organizational maintenance of a site is keeping the navigation system and taxonomy in line with the users’ changing needs.

  With recommendation systems, much of this organizational maintenance goes away. The users organize their own content, in a sense, as the system monitors their constant activity to decide what navigation options go where. What related links could go on a page dedicated to the Beatles? Bob Dylan and Radiohead, for starters. This doesn’t mean that navigation-related decisions go away, however. It still takes a designer to decide what type of information should be displayed on what screen.

Drawbacks

Recommendation systems are not perfect. They have several drawbacks that design teams should consider if they’re thinking about implementing them.

• Difficult to Set Up

  Recommendation systems are intensive, database-driven applications that are not trivial to create and get running. It takes a serious development project to do so. Moving to a recommendation system takes time, energy, and a long-term commitment.

• Maintenance Shifted Elsewhere

  Even though recommendation systems can reduce organizational maintenance, they don’t get rid of maintenance itself. For example, keeping the system up and running becomes a major task, as it is with any significant database-backed system.

• Sometimes They’re Wrong

  Recommendation systems aren’t just a technological challenge. They’re also a social one. Sometimes people are unhappy with recommendations. Sometimes recommendations are wrong. In extreme cases recommendations can be offensive. For example, Wal-Mart got into serious hot water last year when its movie recommendation system recommended movies in an inappropriate way (it still hasn’t put its recommendation system back online.) Similarly, Amazon took some heat when it started cross-promoting its new clothing site by recommending clean underwear to people who were shopping for DVDs.

• Gaming

  Gaming is a big issue, especially on news recommendation sites. Digg, the most popular site of this kind, is continually battling attempts by users to game the system. When a news story is popular on Digg, it gets promoted to the home page as a recommendation for everyone to click on and read. When this happens the clicking and reading can quickly get into the thousands. (Many web sites have actually been brought down by the resulting traffic.) This huge increase in attention is attractive to those people willing to game the system for their own personal benefit. Digg is constantly moderating its members to keep this at a minimum.

Going Forward

Recommendation systems are popping up everywhere, from movies to news to travel and leisure. They provide valuable, personalized information that can greatly influence the way we use the Web. Like any system, however, they are not without their faults. But their benefits seem to outweigh their drawbacks, and they might even be the beginnings of an artificial intelligence for each of us, letting us find our next Radiohead. Recommendation systems are out there: watching, and learning.