is there anything in robots.txt that can speed up the indexing process as opposed to slowing it down ie crawl delay….

I wonder about the order of robots.txt directives for slurp and other robots.

For about a week I have a crawl-delay directive in place for 60 seconds – with slurp coming in very regularly at 2 hits per minute. This is a lot better than the 30 requests per minute I had to deal with before, so obviously slurp obeys the directive at least in part.

Now I wonder about my disallow-directives:
If I have a special Slurp section in my robots txt like this:
—————–snip———-
User-agent: Slurp
Crawl-delay: 60
—————snap————-

then do I have to repeat all allow/disallow commands in this section or will slurp take them from the global “User-agent: *” section?

Also I wonder whether the order of
User-agent: * and User-agent: Slurp sections matters. Given that many spiders have size limits when parsing robots.txt I think that the global section must be the the first with specific User-agents sections following.

Regards,
Ulrich

I set up some mod_rewrite rules to detect several classes of bad requests that would result from failure to recognize a base tag. For the past several weeks, I’ve been logging such requests. Over that time, the vast majority of such requests (over 95%) claim to be from Slurp. I have yet to see a request in this log from Google or MSN.

So, I suspect that either Slurp is sometimes missing base tags or my sites are being visited by an agent spoofing as Slurp.

I’d be happy to share details with a Yahoo! engineer to help track down the root cause. If there’s a Slurp bug, it would be good to identify that. If its a spoofing agent, at least we could start identifying IP addresses to ignore.

Kind Regards,

Paul Loberg
Yahoo! News Search engineer

My site seems to be getting flooded with requests from Yahoo-Newscrawler/3.8-RSS” and “Yahoo-Newscrawler/3.9 RSS”. I was wondering if it responds to ‘Crawl-delay’ in the robots.txt file? I’ve tried and it doesn’t seem to.

It also seems to ignore my disallow. This is what I have in my robot.txt file:

User-agent: Yahoo-NewsCrawler*
Crawl-delay: 5
Disallow: /servlets/

Thanks,
Ed

There´s a lot of 404 that:

http://www.waltercruz.com/ler/graca

but the correct is http://www.waltercruz.com/morningstar/ler/graca

For some reason, the Yahoo is not reading the morningstar part :(

Sorry im sounding like a right pain but as yet we still dont see any action.

Bot action Today, Google Hits 23,000, MSN 8,748 ASK 5,490 Inktomi Slurp 91 hits.

Even Alexa is hiting us harder than your slurp bot.

If you can advise what we should try to do to get your slurp bot to hit us harder it would be much appreciated. We have everything on the pages, links, site map etc, etc yet currently only a small number of pages have been cashed by your slurp bot.

Thanks in advance

Richard Clarke
richc@redgoldfish.co.uk
http://www.redgoldfish.co.uk

Not heard a thing yet from anyone in your sales support. Your bot is still producing a mere 150 odd hits a day at our site with just 723 pages out of 100,000 taken in over a year!.

So far this month c3000 slurp bot hits V 140,000 google bot hits & 40,000 MSN bot hits, even Ask jeeves hits us harder than your bot taking 26,000 hits!.

I just dont get it, We feature high in MSN, ASK, have various high positions in Google yet feature nowhere in Yahoo yet poor sites full of spam and doorway pages dominate the top of the sections where we should feature.

It certainly would be a start if the volume could be some how turned up a bit when your bot visits us.

Any advise you could give us would be apreciated

Kind Regards

Richard Clarke
richc@redgoldfish.co.uk
http://www.redgoldfish.co.uk

No, our crawler doesn’t use etags at this time. It’s something we might consider adding, but most server implementations use etags in addition to the last-modified header

“bot slave”:

You should send feedback to our support staff via the link in blog posting (”feedback, questions or new ideas”) if a robots.txt solution isn’t working for you.

However, robots.txt is the way to opt out… you can disallow our bot altogether, though if the issue is rate of requests, use the crawl-delay tag instead. You may want to re-verify that the bot that is causing you problems is actually Yahoo Slurp as there are other Yahoo crawlers that have different agent names. Otherwise make sure you validate your robots.txt file (try a search for “robots.txt validator”).