« Top Trends in Search in 2007 | Main | Sharing Your Opinion for the Greater Good »
Yahoo! Search Support for X-Robots-Tag Directive to Simplify Webmaster's Control and Weather Update
Today we're announcing support for tags that give webmasters even more flexibility over which pages and documents are crawled and indexed by Yahoo! Search. Specifically, we're extending our support of page level exclusion tags -- NOINDEX, NOARCHIVE, NOSNIPPET, NOFOLLOW -- to provide additional control for archiving and summarization of ANY file type. Previously, these page level tags could only be expressed within html pages through the META directive (for e.g. <META NAME="Slurp" CONTENT="NOARCHIVE">), but based on feedback from our webmasters, Yahoo! now enables these tags to be expressed through X-Robots-Tag directive in the http header, giving webmasters the flexibility to achieve exclusions on PDF, Word documents, PowerPoint, video, and other file types, including html files, and increasing their coverage through a simplified process. Additionally, webmasters no longer need access to html templates in order to express exclusions for html files. To take advantage of this feature, simply add the following page level tags to the X-Robots-Tag directive in the HTTP Header. Here are a few examples:
- X-Robots-Tag: NOINDEX -- If you don't want to show the URL in the Yahoo! Search results.
Note: We'll still need to crawl the page to see and apply the tag, so if you don't wish to have the page crawled, use robots disallow on robots.txt.
- X-Robots-Tag: NOARCHIVE -- If you don't want to display cache link in the search results page.
- X-Robots-Tag: NOSNIPPET -- If you don't want to display summary in the search results page.
- X-Robots-Tag: NOFOLLOW -- If you don't want Yahoo! to crawl links in the page.
Along with this change, we'll be rolling out additional changes to our crawling, indexing and ranking algorithms over the next few days. We expect the update will be completed early next week, but you may see some changes in ranking as well as some shuffling of the pages in the index during this process.
We're at SES in Chicago and WebmasterWorld's PubCon in Las Vegas, participating in a few different panels this week. Please find us if you have any questions or suggestions or drop us your feedback here.
Sharad Verma
Yahoo! Search


Comments
Silicon Valley, in weather updates we typically deploy our enhanced algorithms AND also add high quality sites that are relevant to the query into the search results. In short, we do both.
Posted by: Sharad Verma | December 6, 2007 09:50 AM
So... you're saying that the tags need to be placed in the headers, and they can be displayed in other PDF documents. How would this work for PDFs?
Posted by: uGuX SEO | December 7, 2007 10:31 AM
Is there a difference between the weather reports that say there's fresh data vs. the ones that just say the algorithm has been updated?
Posted by: ttorbes | December 7, 2007 04:44 PM
Is this Update finished yet, because the UK search results seem to have a lot of .ae, .nz, .ca, and .au sites showing up for a lot of search terms.
In fairness, the pages are some of the most relevant I've seen for a long time from Yahoo in terms of the information, there just seems to be a problem with regard to the geo targeting.
Posted by: Quumf | December 13, 2007 04:36 AM
If a page is served with this tag at the HTTP level and also contains the META tag in its HTML content specifying a dfferent set of directives, which set takes priority?
Posted by: - | December 15, 2007 11:56 AM
So according to you. Now web master have to write meta tags which you have mention above in their web page so that the yahoo robots crawl the pages.
Posted by: toindia | February 26, 2008 05:32 AM
Will the pages get indexed with no instructions to the robots?
Posted by: Homebizseo.com | March 12, 2008 05:48 AM
Hello sir, you want to say that we have to include X-Robots-Tag in robots.txt file or we have to make extra file for yahoo spider
Posted by: southall Travel | March 12, 2008 05:49 AM
Please help me to write proper syntax. Is it some thing like below which i have written or its wrong.
If wrong please provide me the sample formate thanks
header(�X-Robots-Tag: index, follow�, true); southalltravel.com
Cheap Flights Dubai
Posted by: jessica | March 18, 2008 07:30 AM
I'm trying to put meta tag inside my index.html instead of creating another html. How I do that and which should served faster, also, if in my index.html have another confirmation meta tag (google), will it work, which one should go first on top of other?
Sorry for quiet a long question, I hope you don't get confused.
Posted by: HonestGain | March 19, 2008 03:23 AM
what you're saying is that we must include X-Robots-Tag in robots.txt file,otherwise,we have to make extra file for yahoo spider ? thank you!
Posted by: tony | March 21, 2008 12:50 AM
It would be desirable to learn more in detail about a practical technique of application of all of it.
Posted by: LeeRoy | March 22, 2008 04:42 AM
"X-Robots-Tag: NOINDEX -- If you don't want to show the URL in the Yahoo! Search results.
Note: We'll still need to crawl the page to see and apply the tag, so if you don't wish to have the page crawled, use robots disallow on robots.txt."
--------------------------------------------------
Does it mean you'll crawl the content too or the robot will stop at the headers level?
Posted by: bihonline | March 29, 2008 05:40 AM
Also (I forgot to mention) why suddenly a complete new name "x-robots-tag"?
What's wrong with the old "robots" name?
And do you then support custom "robots" header which would provide old values, noindex, nofollow,..etc?
Posted by: bihonline | March 29, 2008 08:24 AM
I hope that this news will give further help to those who, like me, hopes to see his sites indexed on yahoo, because until now it is not easy.
Hopefully good
Posted by: Robert | April 1, 2008 01:29 AM
Very convenient feature. I think providing an option to look for recipes in local languages would make this even better
Posted by: kız oyunları | April 1, 2008 01:07 PM
Very important documentation. Great work. Thanks a lot.
Regard.
Posted by: keyfi müzik | April 5, 2008 07:09 AM
what you're saying is that we must include X-Robots-Tag in robots.txt file,otherwise,we have to make extra file for yahoo spider ? thank you!
Posted by: güzel sözler | April 7, 2008 12:25 PM
And do you then support custom "robots" header which would provide old values, noindex, nofollow.
Posted by: you tube | April 7, 2008 12:26 PM
Good to know info. Thanks!
Posted by: Online Tutoring | April 11, 2008 06:42 PM
So according to you. Now web master have to write meta tags which you have mention above in their web page so that the yahoo robots crawl the pages. Please Answer!!
Posted by: industryfinest.com | April 12, 2008 04:24 AM
thank you for the codes..
Posted by: Türk chat | April 18, 2008 09:20 AM
is it necessary for indexing?
Posted by: MaTaDoR | April 20, 2008 01:39 PM
No spamming please. This is no-follow. Yeah. Yahoo always consider such stuff for indexing.
Posted by: Mark | April 27, 2008 12:50 AM
Cool feature. Go on Yahoo!
Posted by: baby scans | May 5, 2008 01:53 AM
Also (I forgot to mention) why suddenly a complete new name "x-robots-tag"?
What's wrong with the old "robots" name?
And do you then support custom "robots" header which would provide old values, noindex, nofollow,..etc?
Posted by: minikperi | May 7, 2008 12:24 PM