Amuse or Awe? Academic Multimedia Grand Challenge for Classifying Images

People love to check out multimedia search on Yahoo!, and we love helping them find what they’re looking for, whether through web, image, or video search. One of the most interesting challenges in multimedia search is how to go beyond plain text and match queries to images so that users can more easily find the image they want.

This was exactly the challenge we posed to the Association for Computing Machinery’s Multimedia Conference held this year in Italy. This competition invites researchers to submit solutions to challenges posed by companies like Yahoo! with presentations that are half elevator pitch and half American Idol.

The winners of this year’s Yahoo!-sponsored challenge are Jana Machajdik, Allan Hanbury, and Julian Stöttinger from the Vienna University of Technology for their proposal to solve “Novel Image Understanding.” Their work, titled “Understanding Affect in Images,” used findings from art history and the psychology of emotion to design a machine-learning system that labels images based on their emotional content.

Text tags associated with an image often tell us about the object in an image but not about its emotional content. The Yahoo! Novel Image-Understanding Challenge was to invent new and useful ways to organize and structure image content. We posed this challenge because we believe that this type of science can eventually offer better results for information and entertainment requests on Yahoo! Search by helping us better organize multimedia content to fit user needs and desires.

Ms. Machajdik developed a simple way to learn what images are considered “cute” or match other descriptions so that queries like “cute kittens” can return appropriate images. She applied her knowledge of the psychology of art and art theory to design features that summarized the color, texture, composition, and content of the image. Using carefully considered emotional categories, she then employed machine-learning techniques to train classifiers. Her system rates images along eight different axes of emotion: anger, disgust, fear, sad, amusement, awe, contentment, and excitement.

Here are some images with analysis from the team’s classifier system:

acm example 1

acm example 2

acm example 3

We’re excited to work with Ms. Machajdik and the broader research community to collectively find ways to bring this kind of science to you and make your multimedia search experiences even better over time. Stay tuned for more from Yahoo! Search on this in the future.

You can read more about her paper and presentation on Ms. Machajdik’s website,

Go to the ACM Challenge 2010 website for more information about the competition, other challenges, and winners.

Kaushal Kurapati and Malcolm Slaney

Kaushal Kurapati, a Senior Director for the Yahoo! Search Product, was one of the judges and Malcolm Slaney, a Principle Scientist at Yahoo! Research, helped organize the competition this year.

  • Subscribe
back to yahoo! search

subscription options

Facebook Fans

latest posts