From Chapter 7 of Chris Anderson's new book The Long Tail: Why the Future of Business is Selling Less of More -- courtesy of Random House:
The catch-all phrase for recommendations and all the other tools that help you find quality in the Long Tail is filters. These technologies and services sift through a vast array of choices to present you with the ones that are most right for you. That’s what Google does when it ranks results: It filters the Web to bring back just the pages that are most relevant to your search term. It’s also what the “Most Popular Tracks” in the acid jazz subgenre on Rhapsody is doing.
Filters make up what Rob Reid, one of the founders of Listen.com, calls the “navigation layer” of the Long Tail. It’s not unique to the Internet and, as he points out, it’s not new:
Interestingly, the power and importance of the navigation layer is not strictly an online phenomenon. For many years American Airlines made more money from its Sabre electronic reservation system (essentially the travel industry’s shared navigation layer for the bewildering world of routes and airfares in the seventies and eighties) than the entire airline industry made collectively from charging people money to ride on planes. From time to time, certain Baby Bells were bringing in more profits from their yellow pages—essentially the navigation layer of all local business before the Web came along—than from their inherited monopolies. And at its peak, TV Guide famously rivaled the actual networks in profitability.
In a world of infinite choice, context—not content—is king.
In today’s Long Tail markets, the main effect of filters is to help people move from the world they know (“hits”) to the world they don’t (“niches”) via a route that is both comfortable and tailored to their tastes. In a sense, good filters have the effect of driving demand down the tail by revealing goods and services that appeal more than the lowest-common-denominator fare that crowds the narrow channels of traditional mass-market distribution.
Reed Hastings, the CEO of Netflix, describes the effect of filters— in this case, sophisticated recommendation engines and ranking algorithms—in driving demand down the DVD Tail on his site.
Historically Blockbuster has reported that about 90% of the movies they rent are new theatrical releases. Online they’re more niche: about 70% of what they rent from their website is new releases and about 30% is back catalog. That’s not true for Netflix. About 30% of what we rent is new releases and about 70% is back catalog and it’s not because we have a different subscriber. It’s because we create demand for content and we help you find great movies that you’ll really like. And we do it algorithmically, with recommendations and ratings.
Hastings believes that recommendations and other filters are one of Netflix’s most important advantages, especially for non-blockbusters. Recommendations have all the demand-generation power of advertising, but at virtually no cost. If Netflix suggests a film to you based on what it knows about your taste and what others thought of that film, that can be more influential than a generic billboard aimed at the broadest possible audience. But these recommendations arise naturally from Netflix’s customer data, and it has an infinite number of “billboards” (Web pages customized for each customer and each visit) on which to display them.
Advertising and other marketing can represent more than half of the costs of the average Hollywood blockbuster, and smaller films can’t play in that game. Netflix recommendations level the playing field, offering free marketing for films that can’t otherwise afford it, and thus spreading demand more evenly between hits and niches. They’re a remarkable democratizing force in a remarkably undemocratic industry.
As we get deeper into filters and how they work, it helps to get an overview of their many types. Let’s start with music. Here are some of the many different filter types a typical user on Rhapsody might encounter in a single session as he or she looks for new music. From the front page, a user might start with categories, which is a form of a multi-level taxonomy.
Let’s say you begin in Alternative/Punk and then choose the subgenre Punk Funk. In that category, there’s a best-seller list, which is led by Bloc Party as I write. If you click on Bloc Party, you’ll find that pattern matching has created a list of related artists, which includes the Gang of Four. A click on that produces a list of “followers” (the Gang of Four created the category of Punk Funk in their first incarnation, in the early eighties), which is a form of editor recommendation (you may also be persuaded by the editorial review).
Among those Gang of Four followers is the Rapture. Click on that, and if you like it, try a custom radio station tailored around that artist, which is a stream of songs by the Rapture and bands that other people who like the Rapture also like, which is a form of collaborative filtering. As you listen to that custom stream, you may find that among the bands that play, the one you like best is LCD Soundsystem. Click on that, listen for a while, and when you hunger for something new, try a playlist that features the band. That, in turn, will introduce you to Zero 7, where you may want to stay awhile.
A half dozen recommendation techniques have taken you from punk to soul, from the middle of the Head to the bottom of the Tail, and every step along the way made sense.
As great as music recommendations are getting these days, they aren’t perfect. One of the problems is that they tend to run out of suggestions pretty quickly as you dig deeper into a niche, where there may be few other people whose taste and preferences can be measured. Another problem is that even where a service can provide good suggestions and encourage you to explore a genre new to you, the advice often stays the same over time. Come back a month later, after you’ve heard all the recommendations, and they’re probably pretty much as they were.
Yet another limitation is that many kinds of recommendations tend to be better for one genre than for another—rock recommendations aren’t useful for classical and vice versa. In the old hit-driven model, one size fit all. In this new model, where niches and sub-niches are abundant, there’s a need for specialization. An example of this is iTunes, which, for all of its accomplishments, shows a pop-music bias that undermines its usefulness for other kinds of music.
In iTunes and services like it different genres—such as rock, jazz, or classical—are all displayed in a similar way, with the main classification scheme being “artist.” But who is the “artist” for classical—the composer, the orchestra, or the conductor? Is a thirty-second sample of a concerto meaningful? In the case of jazz, you may be more interested in following the careers of the individual performers, rather than the band, which may have come together only for a single album. Or perhaps you’re more interested in the year, and would like to find other music that came out at the same time. In all these cases, you’re out of luck. The iTunes software won’t let you sort by any of those.
These are the failures of one-size-fits-all aggregation and filtering. ITunes may be working its way down the Tail, but its emphasis on simplicity—and lowest-common-denominator metadata—forces it into a standard presentational model that can’t cater effectively to every genre—and therefore, every consumer. And this is not to pick just on iTunes—the same is true for every music service out there.
Because no one kind of filter does it all, listeners tend to use many of them. You may start your exploration of new music by following a recommendation, then once it’s taken you to a genre you like, you may want to switch to a genre-level top ten list or browse popular tracks. Then, when you’ve found a band you particularly like, you might explore bands that are like it, guided by the collaborative filters. And when you come back a week later and find that nothing’s changed, you’ll need another kind of filter to take you to your next stop on your exploration. That could be a playlist—catching a magic carpet ride on someone else’s taste—which can take you to another genre, where you can settle in and start the process again.