Since the middle of August 2011, the Bing Webmaster Tools have also been integrating traffic data from Yahoo! into its reports. Please check out the Bing Webmaster blog post for more information.

We have been making great progress in transitioning organic search results to the Microsoft search platform in various markets, as part of the Yahoo! and Microsoft Search Alliance. The Site Explorer website will continue to be available until organic results are completely transitioned to the Microsoft Search platform in all markets (expected to be completed by the end of the year). Webmasters may continue to use the Site Explorer website and all its services until then.

As previously announced, the SiteExplorer YDN API’s are no longer available as of September 15, 2011. We thank all the YDN developer community who used and supported the APIs.

For FAQ’s please refer to Microsoft Webmaster Tools to support Yahoo! Site Explorer Community

– Hemant Minocha, Product Manager, Yahoo! Site Explorer

In an August 2010 blog post, we said we would continue Site Explorer with a focus on new features for webmaster community, even after the transition to Microsoft platforms is complete. We listened to your feedback, and along with the team from Bing Webmaster Center looked jointly at the roadmap for the webmaster tools. Having two webmaster portals for a single source for organic results does not add enough value.  Once organic results are transitioned to Bing in all the markets, we plan to shut down Yahoo! Site Explorer and Microsoft’s Webmaster Tools will be the source for Bing and Yahoo! webmaster site and analytics data.

The Bing Webmaster Center team has been actively adding new features over the past few months, including detailed traffic statistics and a new inbound links feature. Microsoft is also committed to build analytic reports that include Yahoo! search usage. Further details on these reports will be coming soon. You might have some questions that we will try to answer below:

When will Site Explorer shutdown?

We are working with Microsoft to transition organic results in all markets to the Microsoft Search platform. Site Explorer will shut down once all markets are transitioned. This is expected later this year. We will keep updating you with progress of the transition at our Search Blog.

Should I start using Bing Webmaster Center?

Yes, webmasters should start using Bing Webmaster Center.

Does that mean that I should stop using Site Explorer?

No. In a large part of the world, we have not yet transitioned to Microsoft search platform. Hence, it is important that Yahoo! continues to receive your site/sitemap submissions so that search results from Yahoo! are fresh and relevant for markets that have not switched over. In order to continue to receive relevant organic traffic from Yahoo! search results, we encourage you to continue using Site Explorer as you have in the past until we fully switch over to Microsoft search platform for organic results.

Does it mean that I don’t have to optimize for Yahoo! and Bing Search engines separately in the future?

That is correct. We anticipate that later this year, Yahoo!’s organic results will be powered by Microsoft’s search platform in all parts of the world. After that point, any optimization that you do on Bing Webmaster Center tool will be reflected in Yahoo! organic search results.

Will the Site Explorer APIs be shut down?

The APIs were scheduled to be shut down at the end of last year, but the timeframe has been extended for the YDN Site Explorer API up until September 15, 2011.

Note: For BOSS Site Explorer APIs shut down information, scheduled for July 20, 2011, please see this April 2011 blog post.

Will Yahoo! continue to crawl my website?

Although the organic results will be powered by Microsoft Search platforms, Yahoo! will continue to innovate on the search experience beyond the services delivered by Microsoft. In order to provide Yahoo! users with an amazing search experience, our slurp crawlers may continue to selectively crawl websites to help support these efforts.

Hemant Minocha, Product Manager,Yahoo! Site Explorer

Later this week, organic search listings on Yahoo! Search will start to be powered by Bing in the US and Canada (English) markets, and webmasters should begin familiarizing themselves with the Bing Webmaster Center tool.

However, you should continue to provide your site information to Yahoo! using Site Explorer, because in many parts of the world search results will continue to be powered by Yahoo!’s systems until the full transition is complete for all markets by early 2012. This will help to ensure that you continue to get high quality traffic from searches originating on Yahoo! and our partner sites, even in markets that are not yet transitioned to Microsoft’s systems. We will share site information that you provide on Site Explorer with Microsoft during this transition period, to ensure that you get high quality traffic from search results that are powered by Bing also.

Once all markets are being served by Microsoft, we plan to refine the focus of Yahoo! Site Explorer, and webmasters should use Bing’s Webmaster Center to provide information about their website and its structure. Of course, we will notify you well in advance of all markets being transitioned to Bing’s content systems.

Yahoo!’s Site Explorer team is continuing to innovate and provide new features for webmasters. We are planning features that provide richer analysis of the organic search traffic that you get from the Yahoo! network and our partner sites. Stay tuned for more details, and in the meantime we would love to hear from you as to how we can serve the webmaster community better.

Please review these FAQs for more details.

Hemant Minocha
Product Manager, Site Explorer, Yahoo! Search

At the page level, you can now see your SearchMonkey object statistics. This means that if you’ve marked up your pages or provided us a feed that matches one of our existing SearchMonkey templates, your page will display all of the SearchMonkey objects we’ve indexed on that page.

We’ve also included Yahoo! Search keyterms and Delicious bookmark information for this page. Site Explorer gives you a great way to explore all sorts of metadata about your pages, whether it comes from SearchMonkey, Delicious, or our term extraction code.

We now also provide a site-level dashboard that displays additional information, including the number of SearchMonkey feeds you submitted and a history of any custom actions you have requested for your site (such as defining URL query parameters that you want us to ignore).

Finally, we’ve improved some of our backend logic for SearchMonkey feeds so that you can validate small test feeds much faster than before. This enables site owners to iteratively develop their SearchMonkey feeds without having to wait for the batch process to return.

Please try out these new features and let us know what you think!

Yahoo! Site Explorer Team

First, we have updated the xsd files with the XML schemas for Sitemap or Siteindex files. The updated xsd files allow better extensibility of the Sitemaps protocol. All your existing Sitemaps will continue to validate so you don’t need to make any changes.

Second, the protocol will now allow 50,000 Sitemaps per Siteindex, up from the previous limit of 1000 Sitemaps. The file size limit will stay at 10MB.

We hope these changes will make it easier for you to use Sitemaps and submit them to Site Explorer for Yahoo! to index.

Priyank Garg
Yahoo! Search

Today, we launched a new look and feel for Site Explorer (http://siteexplorer.search.yahoo.com/new) that provides a more dynamic interface to accommodate future feature roll-outs. The new interface also includes a new Site Summary page to provide statistics for authenticated sites. On top of this, we’re also increasing the number of rules for Dynamic URL Rewriting that you can enter from 3 to 10.

The new site is located at a special URL to give you some time to play around with it and update your tools that use our interface. We will make this the default experience soon, so please use this time to update your tools. And, as always, please give us feedback on your new experience. We want to hear from you!

Priyank Garg
Yahoo! Search

With everything now in place, the rollout has officially begun. The new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for ‘Yahoo! Slurp,’ though it’ll identify itself as Slurp 3.0 in your web logs.

As the new software undergoes a phased rollout to our production crawlers over the next several weeks, you’ll see the following changes:

a) The crawlers will start crawling from a different and much smaller set of IP addresses, but it’ll still be from the crawl.yahoo.net domain. Any reverse DNS checks to identify our crawler will continue to work. Please note that if you’re using IP-based recognition of our crawlers, you might see a drop in crawl/coverage from Yahoo! We strongly recommend that you move to reverse DNS-based identification of Yahoo! Slurp if you’re using any other method to avoid this problem. The current set of IPs will disappear from your web logs in the next several weeks.

b) The crawlers will also publish a new user-agent, ‘Yahoo! Slurp/3.0.’ Existing robots.txt directives for ‘Slurp’ or ‘Yahoo! Slurp’ will continue to work, but if you have directives specific to ‘Slurp/2.0,’ they won’t be recognized by the new crawler (though usage of the ‘Slurp/2.0′ user-agent is very rare on the web, so you won’t likely be affected). We recommend specifying the shorter version of: User-agent: Slurp. Check out “How do I prevent my site or certain subdirectories from being crawled?” on our Help page for more details.

These changes will affect the main Yahoo! Web Search crawlers. Crawlers that similarly respect the Yahoo! Slurp directive but identify themselves more specifically, such as Yahoo! Slurp China and others, will not be impacted.

Let us know if you have any questions or observe anything unusual.

Sharad Verma & Yoram Arnon
Yahoo! Search

• X-Robots-Tag: NOINDEX — If you don’t want to show the URL in the Yahoo! Search results.

  Note: We’ll still need to crawl the page to see and apply the tag, so if you don’t wish to have the page crawled, use robots disallow on robots.txt.

• X-Robots-Tag: NOARCHIVE — If you don’t want to display cache link in the search results page.
• X-Robots-Tag: NOSNIPPET — If you don’t want to display summary in the search results page.
• X-Robots-Tag: NOFOLLOW — If you don’t want Yahoo! to crawl links in the page.

Along with this change, we’ll be rolling out additional changes to our crawling, indexing and ranking algorithms over the next few days. We expect the update will be completed early next week, but you may see some changes in ranking as well as some shuffling of the pages in the index during this process.

We’re at SES in Chicago and WebmasterWorld’s PubCon in Las Vegas, participating in a few different panels this week. Please find us if you have any questions or suggestions or drop us your feedback here.

Sharad Verma
Yahoo! Search

Try a query in Site Explorer to confirm. As always, let us know your feedback.

Priyank Garg
Yahoo! Search