Comments on: Yahoo! Search Crawler (Yahoo! Slurp) – Supporting wildcards in robots.txt

By: AussieWebmaster

AussieWebmaster — Mon, 13 Nov 2006 15:47:05 +0000

What about a link from another site that contains tracking code… even though the robots.txt file tells the spider not to index the page – it is not coming from the site initially and if it is the arrival page are the rules of the robots.txt file still applied and disallow what is already recorded?

By: Kevin

Kevin — Mon, 13 Nov 2006 12:52:47 +0000

Hello

So if we use /*? will these exclude all urls with query strings?

Pete

By: Jean-Luc

Jean-Luc — Sat, 04 Nov 2006 12:28:33 +0000

You wrote: “Oh, by the way, if you thought we didn’t support the ‘Allow’ tag, as you can see from these examples, we do.”

We need to know the priority rules that apply when “Allow:” and “Disallow:” directives cover the same URL’s (as far as I know, it is not standardized).

Example :
User-agent: Slurp
Allow: /*.html$
Disallow: /backup/

Will /backup/yes_or_not.html be spidered ?

Regarding these priority rules, there is an opinion here ( http://www.conman.org/people/spc/robots2.html#format.directives.allow ) and another conflicting opinion here ( http://books.google.com/webmasters/bot.html#robots ). What about Yahoo’s implementation ?

Jean-Luc

By: yaph

yaph — Fri, 03 Nov 2006 09:12:04 +0000

Google also supports the wildcard characters *, ?, and $.

By: Surya

Surya — Fri, 03 Nov 2006 07:56:05 +0000

Hi
Yes Brian, it will really be helpful if we can get the robots.txt approved from some Yahoo! authority.
What I get from the update is:

User-Agent: Yahoo! Slurp
Disallow: /*.gif$

using this will restrict this url
/public/images/xyz.gif

and allow
/public/images/xyz.gif?sessid=1234asd..

Please let me know if this is what it intends?
And what if I would like to disallow all files in a folder except one?

User-Agent: Yahoo! Slurp
Disallow: /images
Allow: /images/xyz.gif
Just out of curiosity :)

By: Harry G

Harry G — Fri, 03 Nov 2006 07:52:36 +0000

This will simplify our work of writing the robots.txt file.

By: Brian M

Brian M — Thu, 02 Nov 2006 22:36:33 +0000

I agree with Mike and Jean-Luc above. It would be much more helpful if ALL the search engines are going to follow the directives.

Or, you need to at least add the capability to test the robots.txt file for Yahoo directives in your Site Explorer (like Google does for their interpretation of the robots.txt file in their “Webmaster Tools”).

There is already confusion about what works and what doesn’t (as Jean-Luc points out), so how else is a webmaster to be certain that changes will work?

By: Priyank Garg

Priyank Garg — Thu, 02 Nov 2006 22:22:21 +0000

Yes, 'Slurp' also works as the user-agent. Essentially we look for the presence of 'Slurp' in the user-agent string. Please refer http://help.yahoo.com/help/us/ysearch/slurp/slurp-02.html

By: Mike | MetaSearch

Mike | MetaSearch — Thu, 02 Nov 2006 22:00:05 +0000

I think that’ll be very helpful for webmasters if they can use wildcards, but other search engines should also follow if webmasters are really going to use wildcards.

By: Jean-Luc

Jean-Luc — Thu, 02 Nov 2006 20:26:50 +0000

Hi,

I am surprised to see:
User-Agent: Yahoo! Slurp

It used to be:
User-Agent: Slurp

Are both versions of the user-agent understood by all Yahoo! bots ?

Jean-Luc