« Site Explorer Matures a Bit More and Accepts Mobile Feeds | Main | Yahoo! Search Aims for AIIM »
Webmasters Can Now Auto-Discover With Sitemaps
Since working with Google and Microsoft to support a single format for submission with Sitemaps, we have continued to discuss further enhancements to make it easy for webmasters to get their content to all search engines quickly.
All search crawlers recognize robots.txt, so it seemed like a good idea to use that mechanism to allow webmasters to share their Sitemaps. You agreed and encouraged us to allow robots.txt discovery of Sitemaps on our suggestion board. We took the idea to Google and Microsoft and are happy to announce today that you can now find your sitemaps in a uniform way across all participating engines. To do this, simply add the following line to your robots.txt file:
Sitemap: http://www.example.com/sitemap.xml
Please provide the complete URL for your Sitemap on this line. We will pick it up wherever you put it in your robots.txt file. This directive is not specific to user-agent. If you have multiple Sitemaps, you can point to your Sitemap index file on this line. Details about the Sitemaps protocol including this addition are available on the protocol website -- http://www.sitemaps.org.
If you prefer, you can continue to issue Sitemaps to Yahoo! Search by simply inputting the URL for your Sitemap and submitting. Or add feeds to a site you are already managing under 'My Sites' in Site Explorer. This also allows us to provide more feedback to you about what we are doing with the sitemap.
We're also happy to have some east coasters, Ask and IBM, announce their support for Sitemaps. The more the merrier!
We'll also be sharing more this week at SES NY.
If you have other thoughts about how we can collaborate with other search engines on standards such as robots.txt, we'd love to hear from you -- visit our suggestion board.
Priyank Garg
Product Manager, Yahoo! Search


Comments
Excellent idea to point the sitemap in robots.txt. I considered to submit my sitemap via an HTTP request but couldn't found (according with www.sitemaps.org) what is "searchengine_URL" in "searchengine_URL/ping?sitemap=sitemap_url".
Posted by: Grzegorz | April 11, 2007 10:37 AM
Thank you for the info. Followed the directions and was able to do it myself without having to ask anyone. Very simple indeed using the robots.txt file.
Cheers!
Posted by: Jessica Doyle | April 11, 2007 08:38 PM
Is there a reference somewhere to all these non-standard but widely used robots.txt extensions, as well as a validator approved by all the major engines?
Posted by: Marek | April 12, 2007 03:07 AM
great! it's fantastic.it's an advanced and simple way of making your website notable by search engines.
Posted by: pavan | April 12, 2007 06:32 AM
This is great news, and it helps make sure that all of the major search engines provide a standard method of sitemap submission. Kudos!
Posted by: Jon | April 12, 2007 11:29 AM
this might be the most stupid question ever! but! how often should i "resubmit" my site using the yahoo-search tool? daily? weekly?
the context of my xml feed files change daily but i am not sure how often i should resubmit them.
any suggestions is more than welcome.
Thanks
Posted by: Pinsonly.com | April 12, 2007 03:36 PM
do engines take now all sites faster in their index with this system?
Posted by: Paul Buchhorn - Online Marketing News | April 13, 2007 01:54 AM
awesome idea.
Posted by: erik | April 14, 2007 08:23 AM
Good to see more use being made of Robots.txt. How about incorporating other useful hints for search engines such as a geographic location to ensure sites are listed in the correct country search. (This could not be abused as there's only one robots.txt file per site and only one location would be accepted.)
(Also to GlobalOneTechnology, Google's Matt Cutts has blogged on the Webmaster Tools error, it will be fixed.)
Posted by: Stephen Newton | April 17, 2007 07:50 AM
It's good to see uniform recognition for this amongst search engines. If sitemap.xml becomes a regognised standard could search engines simply look for it in the same way they do the robots.txt file?
Posted by: UKMoneyPot | April 17, 2007 02:31 PM
I think yahoo is an awesome search engine! Really appreciate the great work done by you guys! Wish you could take over from goolge!
Posted by: William Clay | April 18, 2007 04:54 AM
Just wondering if what is the reason to force webmasters to provide FULL paths (starting with http:// ) to sitemap files?! Why the "Sitemap:" entry format should NOT comply with "Disallow:" entry format that requires RELATIVE paths, and why there is no way to specify multiple sitemaps in one row like multiple paths to ignore separated by space in the same row?
Posted by: Andrey | April 18, 2007 09:07 AM
Excellent idea - will make life much easier!
It's good that you can sometimes co-operate to do this sort of thing...
Posted by: Rob Scott | April 18, 2007 11:50 PM
I have to agree with the comment made by Andrey, having to specify the URL to the sitemap was a major oversight at the time when this could have done right.
You can't have a sitemap file coming from one server for another server anyway; so what is the point?
Posted by: Alan | April 20, 2007 12:41 AM
Looks like it will still take some time for the robots.text file validators to update, if in fact they do. None of them recognize this new syntax yet.
Just because the big Search Engines now allow this declaration does not in itself mean that it has been added to the robots.txt standard.
Posted by: D C DowDell | April 21, 2007 02:17 AM
Was wondering when this would come along :-)
Great stuff!
Posted by: James Galway | April 23, 2007 02:59 AM
For Webmaster this is the best news from Yahoo, and it's good also feeds submit...
Posted by: turifungia | April 23, 2007 09:51 PM
It's good to see that the search engines have worked together again to get this great initiative completed. Hopefully this will make it easier for Yahoo and the other engines to find good web content.
Posted by: Tim | April 25, 2007 06:01 PM
Good work, thanks!
Posted by: Online Marketing | April 26, 2007 01:02 PM
Looks like the most popular robots.txt file validator: http://tool.motoricerca.info/robots-checker.phtml has updated to support the new syntax. Now this is progress!
Posted by: D C DowDell | April 26, 2007 03:24 PM
Cool, finally I won't have to create multiple sitemap versions.
TedallSearch
http://www.tedallsearch.com
Posted by: Ethical SEO | April 26, 2007 05:31 PM
This is a very good information. I did it on my site and it really helped me. I suggest everyone related to SEO should concentrate on SiteMaps and Robots.txt
Posted by: Nilam Doctor | April 26, 2007 07:32 PM
Great information, this definately help us a lot.
Posted by: Internet Marketing | April 28, 2007 04:18 AM
just off to update all my files. Thanks for the update.
Posted by: Michael | May 1, 2007 03:41 AM
This is a great addition the robots.txt file, maybe search engines will start to follow the other instructions on the robots.txt file
Posted by: Movie Posters | May 1, 2007 03:42 AM
RE: Comment #9: "How about incorporating other useful hints for search engines such as a geographic location to ensure sites are listed in the correct country search?"
We already have that:
- DNS LOC RR (RFC 1876).
- HTML META NAME="Geo.Position" (see http://geourl.org/).
We don't need to clutter "/rbots.txt" with unnecessary stuff.
Posted by: Mr. X | May 2, 2007 04:10 PM
This is amazing, I already started to update robots.txt from my sites. It would be nice to have an automatic site verification using MicroID for Yahoo Site Explorer.
cheers
dimitre
Posted by: dimitre | May 2, 2007 07:14 PM
Thanks, we tried and got excellent result with site map
Posted by: Nishita | May 5, 2007 06:48 AM
great idea, good to see the major search engines working on the same level.
Posted by: Brandon | May 5, 2007 02:10 PM
I use Yahoo sitemap. But curios enough since then I have fallen out ofresults some as 60 pages are not shown. http://www.rosehr.info I`ve been writing to Yahoo Support but I did not receive an answer? What is may wrong doing? Looking forwar hearing from u
Posted by: achim | May 7, 2007 02:47 AM
Hello, I just submitted my site to the site map, how long does this take to authenticate?
Posted by: Gary Richards | May 15, 2007 06:28 PM
FAntastic news.
Posted by: Aden | May 18, 2007 04:30 AM
Thanks for the hard work!
Posted by: Dog Clothing & Dog Collars | May 18, 2007 09:07 AM
how about blogger blogs? and atom.xml?
Posted by: Ben Uy | May 21, 2007 04:40 PM
Yahoo is actually a better search engine than Google, I'm glad some poeple are finally realizing that. This is another great move by the team at Yahoo.
Andy
http://www.tradeshowdirect.com/products
Posted by: Trade Show Displays | May 22, 2007 12:25 PM
Yes i also think that ki yahoo is better search engine but pls tell me how can i add my blog in to yahoo???????
Posted by: Raghvendra Tripathi | May 23, 2007 04:00 AM
Yes, but us YAHOO STORE OWNERS cannot edit our 'robot text files'.
And how come 'site explorer' for my store has NEVER shown all the product pages?
And how come the pages that are shown in 'site explorer' are not crawled for MONTHS at a time?
And how come YAHOO CUSTOMER SUPPORT will not respond to my inquiries on these matters?
Posted by: lilly anne | May 27, 2007 12:36 PM
Finally, the sitemap we have been waiting for arrived thanks.
Posted by: Scented Candles | May 30, 2007 09:55 AM
Thanks for info
but I tried to add sitemap for yahoo to my site www.iptelshop.com and I saw that yahoo didn't like my site, and then I delete sitemap, after that in 7 days was good... I didn't understand, why it happened...if anybody know, please tell me
Posted by: vik | June 11, 2007 03:00 PM
I noticed my site's SERP in Yahoo is twice lower than in Google. I cannot understand why
Posted by: roksolana | June 12, 2007 03:37 AM
Re: Sitemap requiring a completely specified URL
Note that although Yahoo says it must be a complete URL, the definition at sitemaps.org says only that it "SHOULD" be a complete URL, which implies (at least to me) that relative URLs are permitted. If it were a requirement, then the language would have said "MUST." It seems that Google allows and properly handles relative URLs for sitemaps. If Yahoo is saying that they expect only a complete URL, then I say that they are not following the standard agreed upon with the other search engine operators.
Posted by: Mr. X | June 17, 2007 07:11 PM
I am just glad to see major SE working together and agreeing to a standard. This is a good step for content submission to the SEs.
Posted by: George Manlangit | June 27, 2007 05:50 PM
Hi,
I have a 7 year old site that has been in yahoo's directory since 2000/2001 - for some reason yahoo has always just showed 1 url for this domain meanwhile over the years the site has always had around 100 or more pages
I have added the site to yahoo explorer but still it only ever shows 1 listing
My stats shows that the yahoo robot visits the site regular but still only 1 listing
Yahoo currently shows over 6000 backlinks but still just the index page is listed
Ive been racking my brain trying to figure out what is wrong but with no luck
Please can anyone make any suggestions to me as to what to do?
Regards
Reggie
Posted by: reggie | July 2, 2007 02:04 AM
Today pad printing has reached a technical advanced state and the range is quite diversified. Compared to the method described in the historical section not much has changed. The basic procedure has remained the same.
Since the late 1960's an old printing method,which had been particularly well established in the watch-making industry, has been an unexpected boom. Pad printing was discovered for newer, broader applications and with the help of silicone pads and new machine constructions it truly blossomed. Pad printing machine manufacturers sprung up like mushrooms and satisfied the real market need for printing and decorating parts in a simple and inexpensive way. Pad printing allows new designing possibilities for engineers and designers as a result, the products are becoming more attractive and functional.
If you are interested to know more about Pad Printing then please visit http://www.aadish.com.
Best in Quality, Best in service, Best in complete solutions.
We specialize in Machines & Consumables for Surface Decoration-Printing-Marking,Glass Decoration also Pads Plates Cliches inks for pad printing.
At AIM we are committed to providing printing solutions & machines for printing, marking & surface decoration of finished products & components. We have machines & solutions for various products & industries.
AIM brings to you pad printing & customized pad printing solutions from the European Leader, microPrint- Switzerland. Swiss Machines are world renowned for their precision, accuracy & craftsmanship.
The fantastically accurate pad printing machines for printing 1-8 colors & many more from microPrint are now available in India. With our more than 20 years experience in printing industry our experts are best placed to guide you in the choice of right Pad Printer for your application
"Your Product's are valueless, If your brand or logo is not printed clearly on your gift's and promotional articles. Your brand is your asset. Value it."
Pad printing machines, Printing Machine, Printing inks, Pad printing supplies, Glass decoration.
Posted by: Adish International | July 13, 2007 01:31 AM
Excellent.. This is going to make life so much more easier..
Well done Yahoo..
bruce
Posted by: Bruce | July 16, 2007 09:49 PM
I have submitted and authenicated my sitemap more than 3 months ago but the subpages still haven't been indexed by Yahoo. According to the suggestion forum, it appears that I am not the only one having this issue. Any advice would be greatly appreciated.
Posted by: The Dog Clothing Company | July 19, 2007 07:33 AM
This is fantastic - we wish we'd found out about this sooner. So much easier to just click and have Yahoo re-check the site. Thanks.
Posted by: Aquarius Coaching | July 24, 2007 03:01 AM
Finally, the sitemap we have been waiting for arrived thanks.
Posted by: HGH | July 25, 2007 08:36 AM
What about submitting two sitemaps?
One line for each one? or both in the same sitemap: tag?
Posted by: Alberto Garcia | August 3, 2007 12:50 AM
I too am not seeing any index changes based on a sitemap submitted several weeks ago. Is anyone seeing any value in this?
Posted by: wedding favors | August 5, 2007 06:22 PM
I agree this is a long time coming. Finally large database driven sites can get indexed!
Posted by: marketing consultant | August 6, 2007 07:18 PM
Beautiful idea... I've been looking for the simplest way to submit sitemap. Eureka! Can't wait to see the result. Thanks!
Posted by: Rudy | August 11, 2007 05:25 PM
I think its great that these 3 huge companies have come together and started accepting the same file format.
Im sure it will make a huge difference.
Posted by: pune seo services | August 19, 2007 03:14 AM
Hi,
Good now we can use site map easily and effectively.
See my link for cbse results cbse exams results cbse results cbse syllabus cbse sample papers and much more try it.
Thanks
Posted by: prabhat | August 23, 2007 01:25 AM
Very nice. Glad to see that the big 3 search engines are following the same standard.
Joe
http://seemywifestits.com/
Posted by: joe | September 1, 2007 10:00 AM