« An Open Approach to Search | Main | Weather Report: Yahoo! Search Index Update »

February 27, 2008

Making Sitemaps Easier to Manage and Scale

Today, Yahoo!, along with Google and Microsoft, is announcing Cross-host submission for Sitemaps, which will make it easier for webmasters to manage their Sitemap submissions to the major search engines. With this announcement, webmasters can now submit Sitemaps that correspond to several differently-hosted websites using a single mechanism.

For background, a Sitemap file contains the URLs for the pages on a site along with meta-data, such as priority, last crawled date and change frequency for the content. To ensure validity of this metadata, Sitemaps have previously been required to be on the same host and path as the URLs they contain. This requirement forced the Sitemaps files to be hosted on the same servers as the actual site content.

With today's announcement, a Sitemap can now be hosted on a different host and path than the URLs it contains. For example, say you have a Sitemap (sitemap-www.xml) for the URLs on http://www.example.com but you want to put that Sitemap on http://sitemaps.example.com. That is now possible. To make the Sitemap valid and preserve data security you need to refer to it from the robots.txt file on the site where the URLs it contains are located. For example, add the following line to http://www.example.com/robots.txt:

Sitemap: http://sitemaps.example.com/sitemap-www.xml

Our collaboration with Google and Microsoft began back in November 2006 when we announced joint support for the Sitemaps protocol. Since then, we've learned a lot about how webmasters and site owners manage their sites and feeds. We know that segregating user facing content from feeds, like Sitemaps, is important. We've also learned that managing feeds for large websites or websites using third party feed publishing services is critical. We hope this enhancement helps address those needs.

We'll continue to work on addressing the needs of our webmasters through new standards and protocols. If you have other thoughts about how we can collaborate with other search engines on standards such as robots.txt, we'd love to hear from you. Leave us a comment below, or give us your feedback here.

Details about the Sitemaps protocol, including our recent addition, are available on the protocol website. Or, if you're at SMXW this week, bring your questions to our panelists and speakers.


Priyank Garg
Director, Product Management, Yahoo! Search

Sean Suchter
Vice President, Engineering, Yahoo! Search

Permalink | Bookmark | del.icio.us | digg | Filed In: Search

Comments

that's a really good thing cause it was pretty long to submit sitemaps to all search engine. I hope other search engine will join this!!

Posted by: woozor weather | February 29, 2008 10:08 PM

OK, but when are you going to allow the sitemap entry in "/robots.txt" to be a RELATIVE URL - where the domain is taken from the one used to fetch the robots file in the first place?

Sitemaps.org still says that a sitemap declaration in "/robots.txt" SHOULD (not MUST) be an absolute URL, so you've still left this open.

Posted by: - | March 1, 2008 11:40 PM

Sounds like a good deal. We shall wait and see where the charges come from.

Posted by: Chris Heath | March 3, 2008 08:18 PM

This seems to solve one of the problems I have been confronting, but I would prefer to have the sitemap entry in "/robots.txt" to be a URL that I get directly from the robots file.


Posted by: Data Desk | March 5, 2008 07:31 PM

This sounds Great, thanks for keeping us in the loop.

Posted by: VA Refinance | March 7, 2008 12:37 PM

We shall wait and see where the charges come from.
thanks

Posted by: rize, youtube | March 8, 2008 01:54 AM

This certainly simplifies sitemap management for larger sites that are typically powered by many servers. But I still find sitemaps hard to manage for larger sites. It just does not see to scale well.

Posted by: Martin S. | March 9, 2008 11:26 AM

This is great news as it will save much of webmaster's time and energy.Secondly, hosting sitemap on separate servers sounds good.Has this been implemented or we have to wait a bit to make use of this feature????

Posted by: Dr. Sandy at Blogging Guides | March 10, 2008 08:05 AM

This is a great developement, at least in terms of allowing site maps to be hosted on different channels if you have more than one site. I am doing some Search Engine Optimisation for medical plasters and my worry is that having a sitemap located in a different place might compromise ranking.

Posted by: John Kaduwanema | March 11, 2008 02:05 AM

This seems to solve one of the problems I have been confronting

Posted by: luck | March 11, 2008 06:47 AM

I thing, the sitemap is great for SE

Posted by: cooldady | March 11, 2008 06:52 AM

We do it, will see whats going on.

Posted by: Thaishop | March 13, 2008 04:52 AM

I thing, the sitemap is great for SE..

Posted by: youporn | March 13, 2008 07:18 AM

I recently added my site map to Live webmaster tools and when I checked the robots.txt I was invited to add the site map "tag" in the file. It makes a lot of sense and I hope this will help all the SE spider my site faster and better!

Posted by: NoPrescription | March 13, 2008 07:04 PM

If the pages are linked together do we still need a site map.

Posted by: Homebizseo.com | March 14, 2008 02:43 AM

These new sitemaps are so easy to create and implement. Excellent solution.

Posted by: AdmiralAddy | March 14, 2008 12:43 PM

I just wonder if there's any way that we can make such a multiple sitemap in the same domain withing different folder to make the sitemap file size smaller and more efficient. I think if we can implement this, it would be a great benefit for larger domain in managing its domain.

Posted by: Honest Gain | March 18, 2008 07:15 AM

Thanks for making a sitemap easier. You guys are on the ball.

Posted by: Wick-edly Sent Scented Soy Candles | March 23, 2008 01:26 PM

It is easier said than done. I appreciate the great article.

Posted by: va purchase loan | March 27, 2008 07:13 PM

I just wonder if there's any way that we can make such a multiple sitemap in the same domain withing different folder to make the sitemap file size smaller and more efficient.

Posted by: you tube | April 7, 2008 11:33 AM

This is a great developement, at least in terms of allowing site maps to be hosted on different channels if you have more than one site.

Posted by: güzel sözler | April 7, 2008 11:35 AM

This is a great developement,we can make such a multiple sitemap in the same domain withing different folder to make the sitemap file size smaller and more efficient.Excellent solution.

Posted by: Asyari | April 23, 2008 04:14 AM

I hope other search engine will join this!!We shall wait and see where the charges come from.
thanks.


Posted by: Xkobar | April 23, 2008 04:17 AM

I didnt know abt it. This is really AWESOME

Posted by: jonas brothers tickets | April 26, 2008 11:55 PM

Post a comment