SearchMonkey and the structured Web

We’ve just announced an all-new Yahoo! Search experience, with many new features powered by SearchMonkey data.  Since launching our open developer platform in May 2008, Yahoo! Search has enabled thousands of developers to shape the search experience for millions of Yahoo! users. If you are interested in building semantic applications similar to what we’ve come up with at Yahoo! Search, here are some details to get you started.

What structured objects are available?

All of the objects listed on the SearchMonkey homepage are available to you. With the new feature “object refiners,” users can now restrict the search results to specific object types. Site owners contribute data of these objects by marking up their pages with RDF or microformats, or by providing dataRSS feeds. If you’re interested in the actual data of these objects, use the Yahoo! Search BOSS API to request the SearchMonkey data as part of the search request.

How can I access these structured objects?

The SearchMonkey team has been encouraging developers to use our structured data to build semantic Web applications ever since we partnered with BOSS.  Using the BOSS API, you can access SearchMonkey structured objects.

To restrict the result set to pages with SearchMonkey objects, just add “searchmonkey:<objectType>” to your query. The result set from BOSS will only contain URLs that have objects of that type.

For example, the following query returns all of the documents in the Yahoo! Web index that has the words “Sunnyvale” and “pizza” – about 3 million pages.

http://boss.yahooapis.com/ysearch/web/v1/sunnyvale+pizza?appid=wX7OZ3zV34Fy2Y4W4in_vsjFmRhruQNgCxdxn6RUke2c2JVDZdw6bfc1rcEjVnw-&format=xml

But if you only want pages with local business objects on them, you can add “searchmonkey:local” to the query:

http://boss.yahooapis.com/ysearch/web/v1/sunnyvale+pizza+searchmonkey:local?appid=wX7OZ3zV34Fy2Y4W4in_vsjFmRhruQNgCxdxn6RUke2c2JVDZdw6bfc1rcEjVnw-&format=xml

This query returns about 25,000 pages.

Yes, we’ve just thrown out over 90 percent of the result set – but we are after the most relevant results, not simply the greatest number of results. Our new object refiners use SearchMonkey’s structured data to narrow your query from “pizza+Sunnyvale” to actual local business listings within those results. You can use BOSS to retrieve the same structured data and construct any presentation you like.

You can take it a step further and add any of these terms to the query:

• searchmonkey:video – restricts the result set to videos.
• searchmonkey:product – restricts the result set to products.
• searchmonkey:local – restricts the result set to local businesses.
• searchmonkey:event – restricts the result set to events.
• searchmonkey:document – restricts the result set to presentations, spreadsheets, and similar document formats.
• searchmonkey:discussion – restricts the result set to blogs and forums.
• searchmonkey:game – restricts the result set to Flash games.

What don’t I get?

Not all structured data we’ve collected is part of the BOSS API.  For example, some third parties who provide us with feeds have elected to keep that data outside of BOSS. Structured data annotations from technologies built by Yahoo! Research are also not available to third party developers via BOSS. However, we aim to include all data we find embedded in web pages that deploy microformats or RDFa.

Our goal is a successful semantic Web where we extract the semantics as we process Web content. Every page marked up with semantic data makes that much easier for us to extract meaning from that page. And it’s not just us! Google Video Search has recently adopted the same video markup (RDFa and Facebook Share) that SearchMonkey supports.

What’s next?

We will make many more object types available to you soon. In the mean time, you can learn more about SearchMonkey and how we acquire structured data annotations from this new from this post on the YDN Blog.

Kevin Haas

Senior engineering manager, Yahoo! SearchMonkey

Every day thousands of developers drive Yahoo! Search BOSS traffic, serving millions of queries a day. Many of these developers have requested the ability to access Yahoo! Sponsored Search ads to monetize their BOSS innovations. Starting today, in partnership with Domain Development Corp (DDC), our first approved Yahoo! Search BOSS syndication partner, developers can get global access to Yahoo! Sponsored Search results and benefit from revenue generated by their BOSS-powered products.

We invite you to apply to qualify for this program to have Yahoo! Sponsored Search ads appear on your sites. You will need to provide details about your product, including information about your traffic sources, UI framework, and implementation.  Once your product is approved, you will have access to and support for Yahoo! Sponsored Search ads, and you will earn a revenue share.  You can apply at our partner’s site: www.ddc.com.

Please note that in signing up for this Sponsored Search ad program, you will be entering into a contract with a third party, not Yahoo!, and the third party will be providing support and sending you your earnings.  Other Yahoo! Search BOSS Ad partners may be online in the future, so stay tuned for updates.

By now most of you are aware of the Yahoo!-Microsoft search deal, which is still undergoing regulatory review.  While no decisions about BOSS have been made at this point, we will let you know as soon as we figure out the details. These key services will continue to operate for a period of time after we complete migrating our services and technology.

We are proud of the rich community of developers and entrepreneurs who share our enthusiasm for opening search and who continue to amaze us with innovative BOSS implementations. Thank you for your support.

Ashim Chhabra

The Yahoo! Search BOSS Team

Since launching SearchMonkey last May and Yahoo! Search BOSS last July, we’ve reached some impressive milestones: BOSS is now serving 30 million queries a day, and SearchMonkey is celebrating its first anniversary with 70 million enhanced results viewed daily.

Along the way, we’ve made great headway in opening up Yahoo! Search by accelerating the adoption of structured data across the Web and empowering developers to innovate in search. Here’s a look at some of the important initiatives we’ve accomplished with SearchMonkey and BOSS.

With BOSS, we’ve made key updates including allowing developers to monetize through third-party platforms and enabling access to SearchMonkey structured data. The BOSS API is on track to hit 1 billion monthly queries in May, without even including the volume from traditional Yahoo Search syndication. This is more than three times the queries served just six months ago, and ranks ahead of the combined searches on Ask and Facebook, and just behind Microsoft¹.

With SearchMonkey, we’ve launched numerous valuable initiatives over the last year. We turned applications default-on for many sites to improve local , reference, and social searches. We also enabled publishers to more easily display enhanced results for video, games, and documents by adding a few lines of code. Additionally, as a part of our effort to promote a more meaningful understanding of the Web, we have continued to support semantic tech gatherings such as VoCamp.

SearchMonkey is currently live in 23 markets around the world, reaching some significant global milestones:
• 70 million enhanced SearchMonkey results are viewed by users every day.
• Site owners have seen a more than 15% increase in the click-through rates of their SearchMonkey search results when tested against non-SearchMonkey results.
• 200 people enter the developer tool and start creating an application each day.
• Over 15,000 developers have registered to build applications, with over 400 applications available for use in galleries.
• RDFa structured data driven by SearchMonkey has increased by 413% since October, 2008.

In the coming months with SearchMonkey, we will be driving efforts toward increasing structured data on the Web, more uses for existing structured data, and easier ways to display enhanced results for some data types. We’ll throw in a little fun, too, with some open customization of the Yahoo! Search results page. With BOSS, you can look forward to more specialized searches that will help consumers reach their content more easily than ever.

We want to extend our thanks to the developer communities within both SearchMonkey and BOSS and to partners including Facebook, Yelp, and LinkedIn, for joining us in our efforts to make search richer and more open, and for helping us reach these milestones. We’re pleased with what we’ve done in this past year. More importantly, we’re excited about and focused on where we’re taking you next.

Larry Cornett
Vice President, Consumer Products, Yahoo! Search

¹comScore qSearch April 2009. Queries delivered via the BOSS API and served by Yahoo! partners are not counted as Yahoo! Search queries by comScore or other metrics providers.

Today we’re adding several new tools to the Yahoo! Search BOSS developer war chest: Delicious content, advanced language capabilities, and news sorting functionality to the BOSS news service. Like the BOSS Keyterms, Search Monkey, and Site Explorer features released recently, these new additions will empower BOSS developers with more insight into their search results to build unique search experiences.

The immense popularity of Delicious has driven a wave of Delicious tools from Yahoo! and the Delicious community, such as browser add-ons, APIs, and Search Monkey applications. The Delicious data available today through BOSS includes Delicious saves, top tags, and count information.

To obtain this data, use view=delicious_toptags or delicious_saves in the BOSS Web Search call. For example, the following code shows the number of times a search results has been saved in Delicious, its top tags (science, biology, and education), and the count for each tag.

<delicious_saves>12</delicious_saves>
<delicious_toptags>
<tags>
    <tag>
          <name>science</name>
          <count>6</count>
    </tag>
    <tag>
          <name>biology</name>
          <count>5</count>
    </tag>
    <tag>
          <name>education</name>
          <count>3</count>
    </tag>
</tags>
</delicious_toptags>

In addition to Delicious, BOSS is now equipped with Advanced Language functionality so that you can filter specific language results for a given market. Developers can choose to refine language results traditionally, like in this example of default Yahoo! Germany search results, or with strict German filtering to mimic results in this example by adding the strictlang=1 flag to the BOSS Web search call.

BOSS now supports Czech, Hungarian, and traditional Chinese. Language information about Web and news search results is also available by adding view=language in the BOSS call.

Finally, BOSS News Service can now sort by a date or a specified time range of days, weeks, or hours. Please refer to the News chapter of the BOSS Documentation for further information.

If you have any questions or feedback, we encourage you to post them to the BOSS group. We look forward to your continued interest and enthusiasm – there is still a lot more to come.

Ashim Chhabra
Yahoo! Search BOSS Team

The Yahoo! Search BOSS team attended the Alt Search Engines conference in San Francisco earlier this week, and we came away from the conference even more convinced that there’s still a lot of room for innovation in search.  We saw dozens of demos of search engines in various stages of development, many of which use BOSS technologies as part of their approach.

We saw four major opportunities that search entrepreneurs are pursuing:

1)    Vertical Search – Whether it be image, video, health, or even green search, dozens of entrepreneurs are working on building highly focused vertical search engines.

2)    Semantic Search – A group of promising start-ups are focused on using natural language processing or other approaches to build search engines that can understand and respond to the actual meaning of a query.

3)    Discovery Engines – A handful of search engines, such as Kosmix and Worio, are working on building engines that integrate structured data from high quality sources to present users with a starting point to discover the best content on a given topic.

4)    New Presentation Models – Many search entrepreneurs are working on new ways to present search results, by changing the visual presentation or by clustering by topic or data source.

Bill Michels, GM of BOSS, gave a quick overview of BOSS and where we see opportunities to provide technology to the growing search ecosystem. See his presentation on SlideShare:

Big thanks to Alt Search Engines for providing a venue for collaboration and learning.  Keep up the great work, search entrepreneurs!

Graham Mudd
Yahoo! Search BOSS

Today, we’re announcing a handful of new features for Yahoo! Search BOSS as well as important updates on our terms of service and pricing.

Three New Features
Perhaps the most important component of what we’re releasing today is access to SearchMonkey structured data through the BOSS API. The primary way in which SearchMonkey acquires structured data is by using the Yahoo! Web Crawler to scour the web for embedded semantic markup such as microformats or RDF. Starting today, all this data is available to BOSS API users.

The structured data that site owners share with us through feeds will be openly available in the near future if site owners opt to participate. You can read more about how it all works here, but it’s pretty straightforward – just add the “view=searchmonkey_feed” parameter to your API request and we’ll return all available structured data name-value pairs in DataRSS XML. You can also return semantic data in RDF XML using view=”searchmonkey_rdf”.

Here’s an XML example of structured data from President Obama’s LinkedIn page:

We’re excited to see what can be built with this data, so please tag your mashups and products with bossmashup on Delicious.

Second, building on our release of Key Terms last November and SearchMonkey structured data today, we’re also making Long Abstracts available. This is all part of an effort to provide a rich set of document-level data to BOSS developers – in this case a longer description of the page (up to 300 characters compared to 170 previously). You can access these by appending the “abstract=long” parameter to your API request.

Lastly, for years Site Explorer has been a valuable tool for webmasters to understand how Yahoo! Search is indexing their site. Site Explorer also allows users to obtain inlinks for domains and URLs, which are now available through two new BOSS services called se_inlink and se_pagedata.

Open Monetization & Pricing
Effective immediately, we have changed our terms of service to allow developers to use third party monetization platforms (ad-based or otherwise). For obvious reasons monetization is critical to the BOSS ecosystem, so to provide as many opportunities as possible we have decided to adjust our terms to provide developers with more flexibility.

Today we’re also announcing our plans for implementing usage fees for BOSS. We’re introducing fees for a couple of reasons. First and most importantly, we’re hard at work on a number of technologies that will enhance both the functionality and performance of BOSS, and usage fees will help support this development. For example, once we introduce pricing, developers will be able to request 1000 results in a single API call (instead of the current 50). We’ll also be introducing an SLA to ensure BOSS is a robust and stable service for developers. Second, we believe that introducing the proposed pricing structure will improve the ecosystem by optimizing capacity for our serious developers.

You can find all the details on how the fee structure will work here on the BOSS Usage Fees page. Instead of focusing on the particulars, we’ll share the principles we used in developing it. Our goal is to encourage adoption and usage with a low, but fair price – so as not to maximize revenue at the expense of trial and innovation. That is also why we’re going to provide up to 10,000 search queries per day (depending on the type of API call) free of charge to all developers. You’ll notice that the cost to developers is dependent not just on the number of queries requested, but also the type (i.e. how deep your query is). Rather than go with a simplistic “one size fits all” model, we feel that a “pay for what you use” approach is fairest for all types of users.

We’re announcing the fee structure months in advance of it taking effect (likely late Q2 of this year) because we want to give our developers as much advanced notice as possible, and also because we’re as interested as ever in your feedback – so feel free to comment below or on the BOSS developer forum.

Ashim Chhabra
Yahoo! Search BOSS Team

Vik Singh, architect of the Yahoo! Search BOSS team, is the brains behind the BOSS Mashup Framework. Since last summer, he has built many sample apps using the framework. Most recently Vik released an informal mashup (not an official Yahoo! product) called TweetNews, which fuses Yahoo! News with the Twitter API to provide a new ranking model for breaking news queries. Vik took some time to answer our questions about his new application, creative uses of BOSS, and future innovations in search.

Yahoo! (Y!): What’s TweetNews and how does it work?

Vik Singh (VS): TweetNews is a mashup that reorders Yahoo!’s latest news search results based on how popular they are in Twitter. For example, if you search ‘iPhone’ on TweetNews, the service queries BOSS for the latest news results on the ‘iPhone’ and fetches the latest Twitter comments on ‘iPhone’ via Twitter’s API. The service computes how many of these Twitter messages relate to each Yahoo! news result by looking at how much textual overlap there is, then displays the news results re-ranked based on their number of related Twitter messages. Basically this service uses Twitter to determine authority for content that is so fresh it doesn’t have links yet.

Y!: Why did you build it? And what does it demonstrate about what’s possible with BOSS?

VS: One emerging area of search that I think no one has really solved is real-time authoritative search. When breaking news happens (like the Mumbai bombing, Hudson River plane crash, or wildfires), it’s difficult for traditional news sources to discover and prioritize all the information in a timely fashion. It can take several minutes or hours for traditional media to converge on the important stories. However, new social media outlets like Twitter are breaking these important stories faster than traditional media. By looking at the number of users chatting about these topics, one can measure the future newsworthiness of a very fresh story despite its potentially minimal traditional news coverage at that moment.

Although Twitter can be used to discover and rank content, it’s not necessarily the best place to get in-depth, factual information since Twitter messages are very short and unverified. This service ranks actual news stories which provide integrity and in-depth coverage while using Twitter as a signal for ranking. A couple of days ago I searched ‘tweetnews’ on TweetNews and underneath the top result, one of the related Twitter messages described the service as: “The reliability of news with the speed of Twitter.” That pretty much sums up the value proposition of this idea.

I think overall the service was well-received. The traffic filled up my allowed server quotas in minutes. It was quite an honor to read that Wired thought it was the best mashup they’ve ever seen. This idea shows that there are still inefficiencies in search, and that with BOSS, anyone can go out and solve them. All the code and tools I used are open source. It took a little less than 100 lines of code to represent all the search logic, thanks to BOSS. This application couldn’t exist pre-BOSS. Going from nothing to a 100-line search engine is a pretty big advancement!

Y!: What are your thoughts on publicly releasing mashups versus fully “productizing” new ideas before releasing them?

VS: It depends on the idea, but I find that sometimes releasing “quick and dirty” to the world is a great way to test a proof of concept with a real audience. Many ideas that are easy to prototype and test with real users instead go straight through the expensive and slow productization process only to be a dud when they go out to market. In this particular case, releasing TweetNews as a more open third-party mashup provided us invaluable feedback on the idea and empowered our BOSS developer community with more source code and search design patterns.

Y!: Any examples of mashups or search products built using BOSS that you think show potential?

VS: I really like Trogdor, OneRiot, PostRank, InsiderFood, and BuildaSearch, to name just a few. Trogdor saves the users time because they don’t have to press enter or refresh the page; it automatically updates your search results while you type. OneRiot and PostRank aim to solve the social freshness issues that TweetNews highlighted but employ other sophisticated ranking techniques. InsiderFood provides a new user experience for discovering and searching international cuisine.

I like BuildaSearch because it lets anyone build a search engine using BOSS without having to write a single line of code. A user just goes to their site, picks some colors and favorite URLs, and voila, BuildaSearch generates a search engine. I also like many others that have been linked to on this blog.

Y!: Where do you see the most potential for innovation in search?

VS: Blending and personalization of vertical and web search results – I believe vertical search (shopping, local, auctions, news, social) provides invaluable, unique results from what we typically find in Web search. However, users need to learn to pick and choose from multiple search engines based on their query and intent. Wouldn’t it be nice to skip this step and do this automatically for the user? Can we be smart enough to know which search engine is best for that query, and if we’re not exactly sure, can we blend results together from the best verticals and personalize the experience and ranking for the user? I believe every search engine a user encounters should be totally comprehensive, relevant, and personalized based on which site the user is searching from. I strongly believe BOSS can help sites build this new wave of search experiences.

TechCrunch recently announced that voting is open for the Crunchies. Yahoo! Search BOSS was selected as one of the six finalists for the Best Technology Innovation/Achievement category — so we’re asking for your support.

Click the badge below to vote for BOSS!

Thanks!
The BOSS Team

We launched Yahoo! Search BOSS this past July and we just reached a significant milestone – the BOSS API is currently serving more than 10 million queries per day. 10 million in and of itself isn’t particularly significant, but we’re sharing it because we believe growing to more than 100 queries a second in just over 5 months says something about the demand for an open search platform. As a point of reference, the total queries from these developer-built, BOSS-powered search engines would rank ahead of the combined searches done on both Facebook and Amazon, and just behind Ask.com¹. Note that because these queries are delivered via the BOSS API and served up by our partners, they aren’t counted as Yahoo! Search queries by comScore or other metrics providers.

Since launch, we’ve been focused on adding features and building up the ecosystem. We’ll maintain that emphasis in 2009, as well as adding monetizing capabilities to the platform. Our plans have been driven by what we’ve heard from developers on our Yahoo! Group and at six hack days held around the world. We still have a great deal of work ahead, but the feedback we’ve received from the BOSS community has validated our hypothesis that there really was a need for a service like BOSS to help power innovation in search.

If you’re interested in checking out what’s been built using BOSS, here are a few places to look:

• Search Blog Posts on BOSS
• BOSS Mashable Challenge entrants
• Mashups tagged on Delicious
• TechCrunch Search

Thanks to everyone in the BOSS community for helping to make this happen and for being vocal about what we should be working on next. Keep it up!

Bill Michels
BOSS Team

——————————–
1comScore qSearch October 2008

Over the last few months we’ve showcased a handful of innovative mashups that developers have created using Yahoo! Search BOSS. The creations have ranged in functionality and focus, but all have included interesting applications of Yahoo! Search’s index, infrastructure and technology. Today, we’re sharing another useful application of BOSS that TechCrunch just launched – a technology-flavored Web search engine. This new search engine enables TechCrunch users to search for technology-focused articles and company information across both the TechCrunch network of sites and the rest of the Web.

This partnership with TechCrunch illustrates a new BOSS capability called vertical lens technology, which enables partners, often with no search expertise, to create a truly comprehensive vertical search engine that complements their core user experience. Using BOSS, TechCrunch now provides a one-stop shop for all types of search. BOSS not only improves TechCrunch’s standard site search, it also integrates relevant technology content on people, products and companies from across the Web. And if a user conducts a non-tech focused search, they’ll see relevant Web search results.

BOSS vertical lens technology provides a handful of features that were implemented on TechCrunch:

• Real-time indexing of proprietary content

– Once TechCrunch content is added or a user comment is submitted, the search index almost immediately reflects the changes.


• Customized ranking

– The BOSS and TechCrunch teams worked closely to fine-tune the ranking algorithm to fit the TechCrunch audience and user experience.


• Structured search

– BOSS supports faceted refinement around TechCrunch’s structured content. This offers powerful tools for people who want to fine tune their searches or browse TechCrunch content – e.g. narrowing in on articles on Twitter written only by Michael Arrington, or only the most popular articles (popularity is determined by number of comments), or locating all companies that are founded in 2008.


• Blending Web with proprietary content in a single search display

- After querying both the TechCrunch and the Web indices, BOSS blends the results to produce one result set that is both relevant and comprehensive.

BOSS vertical lens is currently available to certain Yahoo! partners – but we’re working to share the technology more openly through the BOSS API. If you are interested in building a search product using BOSS vertical lens technology, please email us here. Our partnership with TechCrunch highlights the beginning of many more BOSS partner launches as Yahoo! continues to open up its search technologies and infrastructure to the world.

Take a look at TechCrunch’s article for additional background and insight.

Stay tuned.

YaJie Ying
BOSS Team