Experts’ domain knowledge improves automated recommendations

9 November 2009 | Ellen Isaacs

I recently started using Pandora, the internet radio that plays music personalized to your tastes, and I’m finding it delightfully eerie. All I did to create my own “station” was enter one of my favorite songs. Now when I let the station play in the background, I keep finding myself  thinking, “Oh yeah! I love that song,” or “That’s a nice song, what is it?”

What makes Pandora so effective and enjoyable is that it:

• Doesn’t limit you to way-too-broad categories (“genre”) or way-too-specific categories (“artist”).
• Doesn’t force you to rate a bunch of songs to determine your taste. Instead, the system asks for just a single song (or album) to recommend a nice variety of highly related songs.
• Bases its recommendations on an extremely rich characterization of the songs it plays. To do so, the people behind the system developed a music-savvy set of attributes, had music experts characterize a large set of songs based on these attributes, and then developed a computer algorithm to group related songs.

This last point is worth emphasizing: Rich characterization based on human experts’ understanding of the domain.

The power of rich characterization

To give you a sense of this sophisticated level of analysis, one of the channels I created consists of songs that, according to Pandora, feature “great lyrics, a subtle use of vocal harmony, major key tonality, electric guitar riffs, and an emotional male lead vocal performance.” Another of my channels features “electronica roots, downtempo influences, rock influences, meter complexity, and headnodic beats.”

I’m not sure what all of that means, but apparently it characterizes music I like.

I’ve often been disappointed by movie sites that allow me to browse only by genre (drama, action-adventure, comedy) or by narrow categories such as actor or director. While Netflix uses a technique called collaborative filtering to recommend movies that people with similar tastes also liked, it rarely suggests movies that delight me the way Pandora so often does. To give a Pandoran-style description, I prefer movies with multidimensional characters that confront complex, believable situations and respond in well-motivated and interesting ways. If a movie has these attributes, I don’t care much about the genre, subject matter, language, or even level of violence. I don’t know how to tell this to Netflix and it hasn’t learned it on its own.

Similarly, I’ve been disappointed by news personalization services that ask you to specify your topic interests based on keywords or generic newspaper sections: international, business, sports, and the uber-vague “lifestyle.” While some do get more specific than this, the keyword approach is limited. Again, here’s my Pandoran-style description of the news I like to read:  follow-up stories about how legislation or business decisions turned out after the data are in. I also like opinion pieces that try to explain the valid aspects of each side of controversial issues and how those views might be balanced. (Hard to find, I know.) And those are just topics — there are other criteria, such as the credibility of the writer and sources cited, quality of writing, degree of sensationalism, relevance to me, and even degree-to-which-I-need-to-be-aware-to-avoid-the-embarrassment-of-not-knowing.

But what category or search term would I enter for these types of stories?

Harnessing expert input in a lightweight way

Pandora does so much better at capturing my tastes because it’s based on the categorization scheme developed through the Music Genome Project. The project identified nearly 400 attributes of songs based on such characteristics as melody, harmony, rhythm, instrumentation, orchestration, arrangement, lyrics, singing, and vocal harmony. The project has a team of 50 musicians who spend 20-30 minutes per song listening and deciding which attributes characterize each song. That’s a lot of effort, but it pays off in providing an exceptional user experience.

In other domains, though, it might not be possible, or perhaps even necessary, to require that much dedicated effort. The hardest part is coming up with the rich set of attributes that capture the deeper aspects of the entities, and that needs to be done by trained experts based on many examples. But once that’s been done, news editors (who are already well-trained experts) could assign those attributes to new stories, allowing readers to point to just a few stories to teach the system the type of news items they want to read regularly.

Many sites already ask people to rate products or services on certain attributes. TripAdvisor, for example, asks people to rate a hotel’s value, rooms, location, cleanliness, and service. That’s a good start, but what I’m proposing is that these systems come up with a richer set of attributes that capture the subtle aspects of what makes for a great customer experience. Once that’s done by experts, you can ask ordinary people (who are already voluntarily writing reviews) to indicate whether the items they are reviewing — hotels, movies, restaurants, gadgets, whatever — have any of those attributes, and then use the ratings to recommend similar items to others. [If there are too many attributes to present at once, people could rate subsets, and then you could aggregate the ratings.]

I realize that this approach runs against the current trend of automating recommendations, which is championed by some of my colleagues at PARC along with many others. Computer scientists and other technologists tend to want to invent ways for computers to not only assign attributes, but also to determine the complex set of attributes that characterize the class.

What Pandora shows us is that we shouldn’t underestimate the importance of using trained experts to help generate great recommendations. I’d like to see that human expertise added back into the process.

Comments (1)