Webmasters Can Now Auto-Discover With Sitemaps
- Posted April 11th, 2007 at 5:00 am by Yahoo! Search
- Categories: Site Explorer
Since working with Google and Microsoft to support a single format for submission with Sitemaps, we have continued to discuss further enhancements to make it easy for webmasters to get their content to all search engines quickly.
All search crawlers recognize robots.txt, so it seemed like a good idea to use that mechanism to allow webmasters to share their Sitemaps. You agreed and encouraged us to allow robots.txt discovery of Sitemaps on our suggestion board. We took the idea to Google and Microsoft and are happy to announce today that you can now find your sitemaps in a uniform way across all participating engines. To do this, simply add the following line to your robots.txt file:
Sitemap: http://www.example.com/sitemap.xml
Please provide the complete URL for your Sitemap on this line. We will pick it up wherever you put it in your robots.txt file. This directive is not specific to user-agent. If you have multiple Sitemaps, you can point to your Sitemap index file on this line. Details about the Sitemaps protocol including this addition are available on the protocol website — http://www.sitemaps.org.
If you prefer, you can continue to issue Sitemaps to Yahoo! Search by simply inputting the URL for your Sitemap and submitting. Or add feeds to a site you are already managing under ‘My Sites’ in Site Explorer. This also allows us to provide more feedback to you about what we are doing with the sitemap.
We’re also happy to have some east coasters, Ask and IBM, announce their support for Sitemaps. The more the merrier!
We’ll also be sharing more this week at SES NY.
If you have other thoughts about how we can collaborate with other search engines on standards such as robots.txt, we’d love to hear from you — visit our suggestion board.
Priyank Garg
Product Manager, Yahoo! Search
- 56 Comments
- Subscribe
Excellent idea to point the sitemap in robots.txt. I considered to submit my sitemap via an HTTP request but couldn’t found (according with http://www.sitemaps.org) what is “searchengine_URL” in “searchengine_URL/ping?sitemap=sitemap_url”.
Thank you for the info. Followed the directions and was able to do it myself without having to ask anyone. Very simple indeed using the robots.txt file.
Cheers!
Is there a reference somewhere to all these non-standard but widely used robots.txt extensions, as well as a validator approved by all the major engines?
great! it’s fantastic.it’s an advanced and simple way of making your website notable by search engines.
This is great news, and it helps make sure that all of the major search engines provide a standard method of sitemap submission. Kudos!
this might be the most stupid question ever! but! how often should i “resubmit” my site using the yahoo-search tool? daily? weekly?
the context of my xml feed files change daily but i am not sure how often i should resubmit them.
any suggestions is more than welcome.
Thanks
do engines take now all sites faster in their index with this system?
awesome idea.
Good to see more use being made of Robots.txt. How about incorporating other useful hints for search engines such as a geographic location to ensure sites are listed in the correct country search. (This could not be abused as there’s only one robots.txt file per site and only one location would be accepted.)
(Also to GlobalOneTechnology, Google’s Matt Cutts has blogged on the Webmaster Tools error, it will be fixed.)
It’s good to see uniform recognition for this amongst search engines. If sitemap.xml becomes a regognised standard could search engines simply look for it in the same way they do the robots.txt file?
I think yahoo is an awesome search engine! Really appreciate the great work done by you guys! Wish you could take over from goolge!
Just wondering if what is the reason to force webmasters to provide FULL paths (starting with http:// ) to sitemap files?! Why the “Sitemap:” entry format should NOT comply with “Disallow:” entry format that requires RELATIVE paths, and why there is no way to specify multiple sitemaps in one row like multiple paths to ignore separated by space in the same row?
Excellent idea – will make life much easier!
It’s good that you can sometimes co-operate to do this sort of thing…
I have to agree with the comment made by Andrey, having to specify the URL to the sitemap was a major oversight at the time when this could have done right.
You can’t have a sitemap file coming from one server for another server anyway; so what is the point?
Looks like it will still take some time for the robots.text file validators to update, if in fact they do. None of them recognize this new syntax yet.
Just because the big Search Engines now allow this declaration does not in itself mean that it has been added to the robots.txt standard.
Was wondering when this would come along :-)
Great stuff!
For Webmaster this is the best news from Yahoo, and it’s good also feeds submit…
It’s good to see that the search engines have worked together again to get this great initiative completed. Hopefully this will make it easier for Yahoo and the other engines to find good web content.
Good work, thanks!
Looks like the most popular robots.txt file validator: http://tool.motoricerca.info/robots-checker.phtml has updated to support the new syntax. Now this is progress!
Cool, finally I won’t have to create multiple sitemap versions.
TedallSearch
http://www.tedallsearch.com
This is a very good information. I did it on my site and it really helped me. I suggest everyone related to SEO should concentrate on SiteMaps and Robots.txt
Great information, this definately help us a lot.
just off to update all my files. Thanks for the update.
This is a great addition the robots.txt file, maybe search engines will start to follow the other instructions on the robots.txt file
RE: Comment #9: “How about incorporating other useful hints for search engines such as a geographic location to ensure sites are listed in the correct country search?”
We already have that:
- DNS LOC RR (RFC 1876).
- HTML META NAME=”Geo.Position” (see http://geourl.org/).
We don’t need to clutter “/rbots.txt” with unnecessary stuff.
This is amazing, I already started to update robots.txt from my sites. It would be nice to have an automatic site verification using MicroID for Yahoo Site Explorer.
cheers
dimitre
Thanks, we tried and got excellent result with site map
great idea, good to see the major search engines working on the same level.
I use Yahoo sitemap. But curios enough since then I have fallen out ofresults some as 60 pages are not shown. http://www.rosehr.info I`ve been writing to Yahoo Support but I did not receive an answer? What is may wrong doing? Looking forwar hearing from u
Hello, I just submitted my site to the site map, how long does this take to authenticate?
FAntastic news.
Thanks for the hard work!
how about blogger blogs? and atom.xml?
Yahoo is actually a better search engine than Google, I’m glad some poeple are finally realizing that. This is another great move by the team at Yahoo.
Andy
http://www.tradeshowdirect.com/products
Yes i also think that ki yahoo is better search engine but pls tell me how can i add my blog in to yahoo???????
Man what an awesome news!
Yes, but us YAHOO STORE OWNERS cannot edit our ‘robot text files’.
And how come ‘site explorer’ for my store has NEVER shown all the product pages?
And how come the pages that are shown in ‘site explorer’ are not crawled for MONTHS at a time?
And how come YAHOO CUSTOMER SUPPORT will not respond to my inquiries on these matters?
Finally, the sitemap we have been waiting for arrived thanks.
Thanks for info
but I tried to add sitemap for yahoo to my site http://www.iptelshop.com and I saw that yahoo didn’t like my site, and then I delete sitemap, after that in 7 days was good… I didn’t understand, why it happened…if anybody know, please tell me
I noticed my site’s SERP in Yahoo is twice lower than in Google. I cannot understand why
Re: Sitemap requiring a completely specified URL
Note that although Yahoo says it must be a complete URL, the definition at sitemaps.org says only that it “SHOULD” be a complete URL, which implies (at least to me) that relative URLs are permitted. If it were a requirement, then the language would have said “MUST.” It seems that Google allows and properly handles relative URLs for sitemaps. If Yahoo is saying that they expect only a complete URL, then I say that they are not following the standard agreed upon with the other search engine operators.
I am just glad to see major SE working together and agreeing to a standard. This is a good step for content submission to the SEs.
Hi,
I have a 7 year old site that has been in yahoo’s directory since 2000/2001 – for some reason yahoo has always just showed 1 url for this domain meanwhile over the years the site has always had around 100 or more pages
I have added the site to yahoo explorer but still it only ever shows 1 listing
My stats shows that the yahoo robot visits the site regular but still only 1 listing
Yahoo currently shows over 6000 backlinks but still just the index page is listed
Ive been racking my brain trying to figure out what is wrong but with no luck
Please can anyone make any suggestions to me as to what to do?
Regards
Reggie
Today pad printing has reached a technical advanced state and the range is quite diversified. Compared to the method described in the historical section not much has changed. The basic procedure has remained the same.
Since the late 1960′s an old printing method,which had been particularly well established in the watch-making industry, has been an unexpected boom. Pad printing was discovered for newer, broader applications and with the help of silicone pads and new machine constructions it truly blossomed. Pad printing machine manufacturers sprung up like mushrooms and satisfied the real market need for printing and decorating parts in a simple and inexpensive way. Pad printing allows new designing possibilities for engineers and designers as a result, the products are becoming more attractive and functional.
If you are interested to know more about Pad Printing then please visit http://www.aadish.com.
Best in Quality, Best in service, Best in complete solutions.
We specialize in Machines & Consumables for Surface Decoration-Printing-Marking,Glass Decoration also Pads Plates Cliches inks for pad printing.
At AIM we are committed to providing printing solutions & machines for printing, marking & surface decoration of finished products & components. We have machines & solutions for various products & industries.
AIM brings to you pad printing & customized pad printing solutions from the European Leader, microPrint- Switzerland. Swiss Machines are world renowned for their precision, accuracy & craftsmanship.
The fantastically accurate pad printing machines for printing 1-8 colors & many more from microPrint are now available in India. With our more than 20 years experience in printing industry our experts are best placed to guide you in the choice of right Pad Printer for your application
“Your Product’s are valueless, If your brand or logo is not printed clearly on your gift’s and promotional articles. Your brand is your asset. Value it.”
Pad printing machines, Printing Machine, Printing inks, Pad printing supplies, Glass decoration.
Excellent.. This is going to make life so much more easier..
Well done Yahoo..
bruce
I have submitted and authenicated my sitemap more than 3 months ago but the subpages still haven’t been indexed by Yahoo. According to the suggestion forum, it appears that I am not the only one having this issue. Any advice would be greatly appreciated.
This is fantastic – we wish we’d found out about this sooner. So much easier to just click and have Yahoo re-check the site. Thanks.
Finally, the sitemap we have been waiting for arrived thanks.
What about submitting two sitemaps?
One line for each one? or both in the same sitemap: tag?
I too am not seeing any index changes based on a sitemap submitted several weeks ago. Is anyone seeing any value in this?
I agree this is a long time coming. Finally large database driven sites can get indexed!
Beautiful idea… I’ve been looking for the simplest way to submit sitemap. Eureka! Can’t wait to see the result. Thanks!
I think its great that these 3 huge companies have come together and started accepting the same file format.
Im sure it will make a huge difference.
Hi,
Good now we can use site map easily and effectively.
See my link for cbse results cbse exams results cbse results cbse syllabus cbse sample papers and much more try it.
Thanks
Very nice. Glad to see that the big 3 search engines are following the same standard.
Joe
http://seemywifestits.com/