Menu

Beta — We’re reformatting our articles to work with the new site, but this is one hasn’t been done yet; it may contain formatting issues. No need to report this. We’re working on it.

The Freedom of Fast Iterations: How Netflix Designs a Winning Web Site

by Joshua Porter
on November 14, 2006

“We make a lot of this stuff up as we go along,” the lead designer
said. Everyone in the group laughed until he continued, “I’m serious.
We don’t assume anything works and we don’t like to make predictions without
real-world tests. Predictions color our thinking. So, we continually make this
up as we go along, keeping what works and throwing away what doesn’t. We’ve
found that about 90% of it doesn’t work.”

NetflixWe were talking with the design team at Netflix. Netflix.com is one of the
most successful web sites in the world: the sole customer interaction point
of their home DVD-rental business. Over the past 9 years, the site has grown
from nothing to serving almost 6 million customers who use the site to prune
their rental queues, rate movies, and handle any billing and transaction issues.
The designers of Netflix.com have a smashing success on their hands, but we
didn’t find them resting on their laurels. They want to get even better, and
for them that means iterate, iterate, iterate.

Netflix isn’t the only company using a fast iterative design approach. Google has also gained attention for their unorthodox design methods, with many people
complaining that they have a huge stable of products, but only a few they’ve
designed well. Yet, Marissa Mayer, Google’s Vice President for Search Products
and User Experience, explains how trying out lots of ideas helped Google design
a better toolbar:

“In the case of the Toolbar Beta, several of the key features (custom
buttons, shared bookmarks) were prototyped in less than a week. In fact, during
the brainstorming phase, we tried out about five times as many key features
— many of which we discarded after a week of prototyping. Since only 1 in
every 5 to 10 ideas work out, the strategy of constraining how quickly ideas
must be proven allows us to try out more ideas faster, increasing our odds of
success.”

This “try and see” attitude is taking hold. The designers at Netflix
told us they try out many new features with every site iteration to keep pace
with the rapidly changing landscape of the Web, as well as their customers’
increasing comfort with the current site. Much of what they do try doesn’t
survive to the next iteration.

So how often does Netflix update its site? Every 2 weeks.

Every 2 weeks they make significant changes. They understand that some of
the changes will work, but most won’t.

At first, this sounds like a frustrating design constraint. In talking with
the team, we realized that it doesn’t frustrate them at all. Instead, it frees
them up to be flexible and adaptive, so they can react effectively to customer
needs. As a result, they don’t deal with the many “when we redesign” issues
that so many of us deal with in the design world. They’re building for the
present — all the time.

Besides living in the now of their web site’s design, The Netflix team gets
other benefits to fast iterations. Here are a few our research has uncovered:

Benefits of Fast Iterations

  • Fail Fast
    A major benefit of fast iteration is you also fail fast. Failing fast means
    you invest less time in the things that don’t work. If you quickly find out
    what works and what doesn’t work, then you take action to turn it into something
    that does work.

    Ironically, teams that fail fast improve as fast, if not faster, than those
    who try to get it right the first time. The reason is simple: Teams trying
    to get it right the first time fail as often as everyone else does. However,
    when they fail, they fail really slowly and struggle to pinpoint problems because
    they’ve changed so much at once, making it harder to identify solutions

  • More Experimentation
    The faster you fail, the more experimentation you can do. You can try out
    ideas that might not have a lot of support, but could be potential winners.
    This allows for an innovative environment.

    Perhaps you’ve heard of Google’s 20% time? They expect their engineers to
    work 20% of their time on a personal project — an experiment they find personally
    interesting. This program has the effect of bootstrapping experimentation,
    so it will happen more often.

  • Learn Quickly
    We’ve all had the experience of sitting in meetings arguing whether something
    will work or not. Usually, both sides just don’t have enough data to go on,
    and they end up going with their gut or with the loudest arguer (for better
    or worse). Fast iteration helps solve this problem by giving developers a platform
    on which they can test quickly, helping to collect data about any outstanding
    questions instead of resorting to opinionated arguments.
  • Provide Continuing Interest
    In addition to improving your design, fast iterations may have a psychological
    effect on users. Those users who use your app with any frequency will notice
    the changes, and if the good ones stick, they’ll appreciate your ongoing efforts
    to improve.

    The best teams not only design the changes, but design the process for introducing
    the change. They experiment with methods to overcome the users’ natural resistance
    to change, providing migration paths and clear benefits for each improvement.

  • Reduced Risk
    Quickly iterating helps reduce risk during design. If teams can make many
    small changes instead of a few larger ones, they mitigate risk because they
    know which changes have what effect. If a design team makes many changes at
    once, they have a harder time knowing which parts work and which parts don’t.
    When you make only one or two changes at a time, you know immediately what
    effect it has. Reducing risk is a valuable outcome of moving to fast iteration.

Side Effects

These benefits don’t come easy. There are significant changes design teams
have to make to their core process to iterate quickly. It’s not a switch a
team can turn on or off.

  • Culture Change
    Most designers are accustomed to long release cycles. Fast iteration and fast
    evolution of design creates a different kind of design environment.
    Gone are the grand visions of the redesign, where teams spend months retooling
    vast areas of the site. Replacing it is the idea that the site is a living,
    breathing design that needs constant care and attention. The team at Netflix
    calls themselves “compulsive data wonks”. They rarely dream very
    far in the future. Instead, they’re concerned with what’s happening right now.
  • Design Determinism
    When teams make the switch to fast iteration, it changes the site’s testing
    methodology. Testing becomes ongoing. After a release, you test for a certain
    period of time to determine what to keep and what to throw away. Then you start
    the process over again immediately. And repeat.

    To some designers this sounds overly deterministic: Doesn’t this take the
    fun out of design? If all the decisions are cut-and-dried, what does that say
    about creativity? What about longer-term effects? Is it possible that some
    features take longer to catch on than others, and that an early flop might
    not mean it’s not a valuable feature? With fast iterations, if the feature
    doesn’t work now, then it’s not right for the site, no matter how creative
    it is.

  • You’re Either With Us…
    Netflix’s Chief Talent Officer, Patty McCord, told us their process of fast
    iteration causes uncomfortable situations for some designers. Once, a designer
    had spent time and energy working on a feature that testing showed didn’t work.
    When it came time for the team to remove the feature from the site, the designer
    was distraught. He had become too emotionally invested in his design, and it
    got in the way of his job. He ended up parting ways with the team and moving
    on. Unfortunately, the process of fast iteration affected more than the product
    itself.

Release Early, Release Often

Eric Raymond, in his famous work, The Cathedral and the Bazaar, wrote the
words organizations like Netflix live by: release
early, release often
. At the time, Raymond was referring to software
created in the open commons, by thousands of people, with little or no delivery
deadline.

This dictum now applies equally well to web-application development. Fast
iterations provide the freedom to innovate because teams can test more features.
This reduces the risk of each tested feature. However, it takes getting used
to and not everyone adjusts so easily. As with most design processes, teams
get the most benefit from quick iterations when they fully embrace the process.