Yahoo! Search Blog: Yahoo! Search Crawler, Slurp, is moving

« Yahoo! oneSearch - Redefining What You Can Do With Your Phone | Main | The Power of Community at ETech »

March 28, 2007

Yahoo! Search Crawler, Slurp, is moving

Anyone who has seen their web server logs, has seen the Yahoo! crawler come by and leave its mark. It identifies itself as 'Yahoo! Slurp' and its domain is inktomisearch.com. For those of you who follow the search space, you can easily guess that this is one of the last historic remnants from our acquisition of Inktomi a few years ago. Well, the crawler finally decided to move and find a new home. We are moving our crawler from inktomisearch.com to crawl.yahoo.net.

The user-agent will continue to be Yahoo! Slurp, hence you do not need to make any changes to your robots.txt file. Nor are we changing the actual IP addresses of the crawler infrastructure during this transition. However, if you do any reverse DNS checks for the crawler identity or have any network access rules to allow inktomisearch.com, please also update them to allow for crawl.yahoo.net.

The crawlers will be switched over in phases starting immediately. You'll see an increasing number of Yahoo! Slurp visits from crawl.yahoo.net and fewer and fewer from inktomisearch.com during the transition, which will take a few weeks.

Keep an eye on this space and the Yahoo! Search Blog for more details on when the migration is complete and details on bot verification with the new domain name.

Priyank Garg
Product Manager, Yahoo! Search

Permalink |

Bookmark |

del.icio.us |

digg | Filed In: Misc

Comments

Thanks for the update!

Posted by: Mark Barrera | March 28, 2007 08:09 AM

Thanks Priyank, are there significant changes apart from the change in domain name?

Posted by: Elmer W. Cagape | March 28, 2007 06:18 PM

great stuff I glad I found out that! good luck with the change yahoo!

Posted by: domain inform | March 28, 2007 07:56 PM

Thanks for the information.. It's very useful.

Posted by: Amit Doda | March 28, 2007 10:00 PM

Thanks a lot for the information.

Posted by: Autoverzekering | March 28, 2007 10:48 PM

thanks for the update, i have blogged it in.

Posted by: mark | March 29, 2007 01:43 AM

Hi, good to know.

Is it possbile to know why this change? Something ralated with new crawling system?

Posted by: simleon | March 29, 2007 01:56 AM

Thanks for information. Hope that information current improvement or change will be available continue.

This is OK that "user-agent will continue to be Yahoo! Slurp". Otherwise lots of seo have to change their website's robot.txt file.

Posted by: Suresh Chowhan (SEO) | March 29, 2007 04:03 AM

Thanks Priyank

Thanks for the info.

Posted by: Nirupam Roy | March 29, 2007 10:24 PM

Does that mean you'll be fixing Slurp so it won't attack my site like a DOS attack anymore? Currently, I have hundreds of Slurps attacking my servers. Pretty much business as usual.

I like my site being indexed, but certainly you can do better. Google does it with just a few bots at a time. And, Google does a far, far, faaaaar better job of indexing the site with less resources.

Posted by: Peter Davis | March 30, 2007 07:09 PM

Thanks for heads up!!!

Posted by: Tilak | March 30, 2007 10:23 PM

I searched a long time for such an great article. Thank you

Posted by: gutschein | April 8, 2007 06:04 AM

Thank for the Info !

Posted by: Julie | April 12, 2007 04:07 PM

Yes truly great article

Posted by: massive penis | April 25, 2007 05:56 AM

Thanks for the info!

Posted by: Home Furnishings | May 21, 2007 11:49 AM

Thanks sir!