This was exactly the challenge we posed to the Association for Computing Machinery’s Multimedia Conference held this year in Italy. This competition invites researchers to submit solutions to challenges posed by companies like Yahoo! with presentations that are half elevator pitch and half American Idol.

The winners of this year’s Yahoo!-sponsored challenge are Jana Machajdik, Allan Hanbury, and Julian Stöttinger from the Vienna University of Technology for their proposal to solve “Novel Image Understanding.” Their work, titled “Understanding Affect in Images,” used findings from art history and the psychology of emotion to design a machine-learning system that labels images based on their emotional content.

Text tags associated with an image often tell us about the object in an image but not about its emotional content. The Yahoo! Novel Image-Understanding Challenge was to invent new and useful ways to organize and structure image content. We posed this challenge because we believe that this type of science can eventually offer better results for information and entertainment requests on Yahoo! Search by helping us better organize multimedia content to fit user needs and desires.

Ms. Machajdik developed a simple way to learn what images are considered “cute” or match other descriptions so that queries like “cute kittens” can return appropriate images. She applied her knowledge of the psychology of art and art theory to design features that summarized the color, texture, composition, and content of the image. Using carefully considered emotional categories, she then employed machine-learning techniques to train classifiers. Her system rates images along eight different axes of emotion: anger, disgust, fear, sad, amusement, awe, contentment, and excitement.

Here are some images with analysis from the team’s classifier system:

We’re excited to work with Ms. Machajdik and the broader research community to collectively find ways to bring this kind of science to you and make your multimedia search experiences even better over time. Stay tuned for more from Yahoo! Search on this in the future.

You can read more about her paper and presentation on Ms. Machajdik’s website, imagemotion.org.

Go to the ACM Challenge 2010 website for more information about the competition, other challenges, and winners.

Kaushal Kurapati and Malcolm Slaney

Kaushal Kurapati, a Senior Director for the Yahoo! Search Product, was one of the judges and Malcolm Slaney, a Principle Scientist at Yahoo! Research, helped organize the competition this year.

In August and September 2010, we conducted exploratory studies at our Bangalore and Taiwan usability labs to help us better understand people’s behaviors, intentions, and consumption of multimedia search from any provider.

We wanted to answer some key questions about multimedia search: How do people search for multimedia online? How do people consume the image or video results that they get through multimedia search? And what do people like or dislike about the experience of searching for videos and images online?

User experience research at Yahoo! is accomplished primarily through direct observation of use situations – either in the lab or in natural environments like people’s homes, offices or internet cafes. Our labs are fully equipped, so design researchers can conduct lab based studies like user interviews, usability tests, and focus groups.

In the multimedia user research, we learned some interesting insights:

• Though it is difficult to pinpoint a ‘trend’ in image search topics, we found that searches for movies and celebrities emerged as popular topics across the countries where we conducted the research. (India and Taiwan).
• In India, we found that visual thumbnails are a key ‘hook’ in image search. Other information like resolution and URL are secondary influencers that people might look at, but only after the image thumbnail.

During the research we also got a lot of feedback about the new Facebook integration’ feature that we launched recently. This feature allows users to search for albums that friends and family have posted on Facebook, through Yahoo! image search. A lean back “theater mode” experience enables users to enjoy the high quality images. To use this feature, users need to link up their Yahoo user ID and Facebook ID as a onetime process. To see and use this feature, type a search query on Yahoo! Image Search, then scroll to the end of the image results page. Currently this feature is available through http://images.search.yahoo.com/
In the exploratory study we did in Bangalore, most participants reacted positively and enthusiastically about integrating Facebook to image search:

• Users liked being able to view an entire ‘album’ versus individual pictures related to a search topic. They told us that it was more holistic and saved them the time spent to search or do multiple searches in order to create the ‘whole picture.’
• Users felt that Facebook image integration adds a familiar face and contact point to image search results, making the experience more personally relevant and interactive.
• Most users preferred to see all search results on a single long page and scroll rather than go from page to page. People say it’s more helpful to display all the images on the same page so they can quickly get to the things they’re interested in on the page, without needing many refinements to their searches.

In the exploratory study in Taiwan, our study participants felt that it was interesting to find Facebook photo albums from their friends, but some users asked questions about privacy.

We would love to hear from you if you use multimedia search, such as Yahoo! Image Search or Yahoo! video search. If you have some feedback on what you liked or disliked about your experience searching for  images or videos, please share them with us using the ‘Comments’ option below.

We are also constantly looking for participants in our user studies. If you would like to influence the Yahoo! user experience by sharing your opinions and feedback, sign up to participate in user studies based in India or the U.S.

Devika Ganapathy
User Experience Researcher – Yahoo! India Insights

Highlights of Yahoo! speakers at SES:

Tuesday, August 17

Search: Where to Next?
11:30 a.m.–12:30 p.m.
Speaker: Shashi Seth, Senior Vice President, Search Products

Digital Asset Optimization
1:30 p.m.–2:30 p.m.
Speaker: Josh Cobb, Sr. Director, Americas/Business Development & Partnerships Group

Price-per-Click (PPC) Lab
3:00 p.m.–5:15 p.m.
Speaker: David Roth, Director of Search Marketing

Developing Great Content
4:15 p.m.–5:15 p.m.
Speaker: Wendi Sturgis, Vice President, North America, Business Development and Partnership Group

Wednesday, August 18

Crossing the Digital Divide: The Leap from Search to Display
11:00 a.m.–12:00 p.m.
Speaker: Dave Zinman, Vice President and General Manager, Display Advertising

Getting Mobilized Marketing Strategies
1:00 p.m.–2:00 p.m.
Speaker: Paul Cushman, Senior Director, Mobile Sales Strategy

Thursday, August 19

Social and the Marketing Mix
2:00 p.m.–3:00 p.m.
Speaker: Bobby Figueroa, Vice President, Product Development

Advanced Paid Search Tactics
4:45 p.m.–5:45 p.m.
Speaker: Jon Mette, Senior Strategist, Search Optimization and Strategy

See the complete SES San Francisco conference agenda.

The Yahoo! Search Marketing Blog will be tweeting (#sessf), posting to Facebook, and uploading snaps to Flickr page. So check back here for updates. We hope to see you there!

Yahoo! Search Blog: We’ve been hearing a lot about moving away from the 10 blue links and providing exciting new search experiences. Can you tell us more about how Yahoo!’s search sciences are contributing to that effort?

Ben Shahshahani: The whole concept of search being just a way of finding the top ten relevant articles has been changing for some time. What you see now on the search results page now is a very rich blend of information coming from different sources, like back-ends of local databases or news, Twitter, real time, social information, images, and videos.

Now, the other thing that has been happening is an integration of structured data and unstructured data, so structured meaning that there are particular attributes to different entities. We have a pretty active technology and science effort in trying to understand the main object, attributes, and relationships – not just the text on a web page. So, rather than actually having to go through various content pages and trying to find that information, we actually bring that information up front for the users.

Q: What can users look forward to with new developments in determining user intent?

A: We start very early in the process, trying to basically read the mind of the user and figure out what the user might be asking for so we can help them get to the information they want very quickly. So if you think of Yahoo!’s search assist, the minute you start typing the first few letters, the tray opens and we start making recommendations for the completion of the query.

Recently, we made a change where we promote time-sensitive queries in our Search Assist feature. For instance, if we think that an event that’s happening right now will be important to users, we may suggest the latest news about that query. So, if the World Cup is happening, and you start typing in “w-o-r,” you’re going to see queries that are related to World Cup instead of a less timely suggestion like “world map”.

Once a query comes in, the question is: “what is the intent” or “what are the common intents of the users submitting this query?” To answer that question, we use a variety of ways to understand the query – a lot of the queries are about objects.

Q: What do you mean when you talk about objects in the search context?

A: Objects are things in the real-world.  They can be events, a location, a person or a product. Our active effort in understanding attributes and their relationships helps us find out the things you can do with those objects. Analysis of the data that we have tells us the kinds of things people do after they search for something. This helps us identify the possible intents when a user submits a query about an object. Now, on the other end of the spectrum, we are really looking towards trying to guess the intent of the user even before they start submitting a query, to give them the information they need before they even think to search.

Q: What are some trends you have observed about data sources and how users are getting content?

A: If you look at the way people discover content and information, we also see a change there. Nowadays content is often discovered through social networks. Your friends or people that you follow on Twitter may share a URL and as a result the traffic to those sites is going up.

Another trend has been that search is generated in the context of some other activity. For example, you may be reading an article, and that article might contain references to some things or objects that trigger your interest. We may extract information from our backend and then provide a blend of structured and unstructured data on the side for the user to see.

Q: Let’s talk about search and social. Now that we have more social data available to us than we did a few years ago, how are we making use of that data to determine search intent?

A: Social is a more recent topic. There are multiple facets into it. On one hand, social is a kind of data. We have social and real time content that is generated on Twitter, on Facebook, on Yahoo! Pulse, the comments that people leave, the traces that they leave behind on various pages. To the extent that these are all content, in the context of search, you want to present the most relevant content to a user, including social content. We already trigger results from Twitter when you do a search. But we have algorithms that decide whether or not reaching out to Twitter as a source of content is interesting for that query, for that person.

The other aspect is insights users leave behind by their social activities. The comments you leave, by sharing, by “thumbs up” and “thumbs down,” and the various other kinds of sharing you do online all help in enriching our insights about users and content.

Q: How does your research at Yahoo! Labs influence the way you view search?

A:  One focus area for us is really about understanding users in different contexts. Search is the context where the intent is the most explicit. But even in the context of search, we like to consider search not as a stateless, information-extraction, but sort of an ongoing dialogue between the user and the system. User intent even in the context of search can be beyond the query that the user submitted; it could be in the context of the entire session. That’s from the perspective of understanding the user. We also have a strong effort in understanding content. What are contents about, in terms of the objects and relationships associated with the query? Bringing these two together can create awesome experiences for users.

Yahoo! has been at the forefront of content discovery since our early days as “Jerry and David’s Guide to the World Wide Web.” Today we provide a unique blend of the most popular and most effective types of online discovery – from editorial, communications, and search to newer social sharing. Every day, people are sifting through the massive amount of information on the web and exploring news, videos, images, and other information on Yahoo! Search technology is becoming more critical to helping people find the stuff that interests them most, no matter where they are online. Today, tons of searches can actually be done without ever hitting return on a search box – for example, by checking out our contextual shortcuts for underlined words while you’re perusing the latest news, or using cool new mobile apps.

The industry as a whole is changing rapidly, and as search continues to evolve, so will the measurements used to understand the market. People no longer search to find a list of blue links; they search to find answers in the shortest amount of time possible. We believe that surfacing the right information at the right time is more important than the number of total results delivered or number of traditional queries conducted.

As we push forward with new search features to bring relevant search results to more people in more places, we look forward to talking with folks like comScore and the industry at large about how to measure the new paradigm of search. Our goal is to help evolve measurement standards, definitions and metrics so the industry has what it needs to accurately understand trends in search share across different types of searches and different companies.

What do you think is the best way to measure the success of a search provider – share of queries, share of query type, share of revenue, engagement, consumer loyalty, search sessions per month?  There is no single answer to this question, but I expect that as the industry evolves, we will identify a blend of metrics to better meet the needs of all audiences who are interested in what’s happening throughout the search industry.

Shashi Seth
Senior Vice President of Yahoo! Search Products

When “American Idol” is on, it’s a busy day for a trend spotter like me.

I sometimes sport the fancy title of Web Trends Analyst: My job is to interpret search interest in pop culture and help develop news stories for reporters and readers like you. As you might imagine, a billion or so searches lend themselves to many narratives. Figuring out the “what-does-it-all-mean” goes beyond declaring the winner in an ever-changing popularity contest, or what’s on top of everyone’s mind day to day.  (Although, I can tell you Sandra Bullock has lately beat out Lady GaGa as the most-searched celebrity, and “Twilight” leads in searches for the most anticipated movies of the season.)

Instead, I face questions like this: What does the rise in apocalypse-related searches following natural disasters say about our modern society? Are the lookups following Tiger Woods’ story prurient, or are we repeating our ancient fascination with the morality tale? And can search activity project what the masses will decide, even before the masses know themselves?

Fear not, I harbor no aspirations to be the next Nostradamus (who has measured in the top 30,000 searches in the past 30 days). While you can use search trends to gauge fan interest, you can’t use queries to predict who’ll win an Oscar or the World Series.

But queries can point to some interesting projections when it comes to reality shows, where the fate of a limited set of contestants is decided by a mass audience. American Idol presents an increasingly rare and unique confluence of such circumstances (as does “Dancing with the Stars” to some extent, but the judges’ scores also play into the ABC show’s formula.) By gauging who the core “Idol” voters are (generally ladies 30-54), determining the influence of hometown loyalties, and factoring in the tween factor, we’ve made 3 out of 4 correct projections. (Darn you, dark horse Kris Allen and America’s propensity for the underdog, even among underdogs.)

All the interpretative dance we do around search data is actually more important than figuring out the winner. Analyzing search trends helps us understand the impulses and processes of why people make their choices at that particular moment in time. About the only other comparable phenomenon is, well, the presidential elections. Back in April 2007, when party candidates were just raising money for their presidential campaigns, I ranked the Democrats and Republicans by Search popularity. The order adhered pretty closely to the amount of money they brought in. Back then, a largely unknown Barack Obama edged out Hillary Rodham Clinton, whose first-quarter fundraising was separated only by a mil.

Who we are, as a collective culture, is what fascinates me. As a former features and entertainment reporter, I love having at my fingertips the curiosity of the masses (anonymous and in aggregate). The typical trend story relies on a handful of interviews, and a (hopefully) savvy reading of repeating occurrences or underlying themes. A better trend story pinpoints which specific groups of people practice the trend and maybe throws in some robust poll data or study results. The best ones recognize the historical context in which the trend has (re)emerged and gets to the “why.”

For me, I’m getting to the why. My job is to read the Search tea leaves, distill the narratives that fit the fact and context, and serve up my brew. You can get a sip of what we trendspotting editors do throughout Yahoo! from the Buzz Log, our annual Year in Review and on Twitter via Yahoosearchdata. Keep searching — I’ll be looking out for all of you.

Vera H-C Chan
Senior Editor
Yahoo! Buzz

We mentioned yesterday that Yahoo!’s Tom Wailes took part in a panel discussion called “The Big Picture from the Big Players” at the O’Reilly Where 2.0 conference. The panelists talked about many interesting opportunities in local search, including user participation, social search, and mobile search in the local space. Check out the video to see Tom’s recap of his discussion on where local search is headed and the major themes that evolved from the conversation among Yahoo!, Bing, Google and Yelp.

Wednesday, March 31
Time: 2 p.m. PT
Speaker: Tom Wailes, Sr. Director, Local Product & Design
Panel: The Big Picture, from the Big Players

In this session, representatives from the major search engines discuss the state of where we are and where things are going in the future.

We hope to see you there!

Check out the video below to see what Yahoos were discussing with people on the show floor. In the video, Rob Wilk , National Director of Search Optimization & Strategy, introduce his panel on Advanced Paid Search Tactics and David Miller talk about his session on Ads in a Quality Score World, as well as what he sees to be the benefits of attending the show.

Other panels we participated in included:

• Digital Asset Optimization, Josh Cobb, Sr. Director, Americas | Business Development & Partnerships Group, Yahoo!

• Search: Where to Next?, Larry Cornett, VP, Consumer Products, Yahoo! Search

• From Real-Time Search to Dynamic Discovery, Larry Cornett, VP, Consumer Products, Yahoo! Search

• Meaningful SEO Metrics: Going Beyond the Numbers, Dennis R. Mortensen, Director of Data Insights, Yahoo!

• Developing Great Content, Wendi Sturgis, VP, Partner Management Group, Yahoo!

Tuesday, March 2, 2010

Time: 10:45 a.m. – 11:45 a.m.
Panel: Real Time Search & The Major Search Engines
Speaker: Ivan Davtchev, Senior Product Manager, Yahoo! Search

Wednesday, March 3, 2010

Time: 1:30 p.m. – 2:45 p.m.
Panel: Search Meets Display, Display Meets Search
Speaker: David Oliveira, Regional Vice President Sales, Yahoo!

Time: 4:45 p.m. – 6 p.m.
Panel: Ask The Paid Search Reps
Speaker: Tomaso Pozzi, Product Manager, Yahoo!

Thursday, March 4, 2010

Time: 9 a.m. – 9:45 a.m.
Panel: Keynote – The State Of The Search Union
Speaker: David Roth, Director of Search Engine Marketing, Yahoo! Inc.

Time: 10 a.m. – 11:15 a.m.
Panel: Bringing SEO In House: How To Be Successful!
Speaker: Laura Lippay, Director of Technical Marketing, Yahoo!

Time: 11:30 a.m. – 12:30 p.m.
Panel: Ask The Search Engines
Speaker: Arnab Bhattacharjee, Senior Director, Yahoo! Search Technology and Engineering, Yahoo!

We’re looking forward to seeing you there!

Yahoo! Search Team