Yahoo! Search Crawler, Slurp, is moving

  • Posted March 28th, 2007 at 8:00 am by Yahoo! Search
  • Categories: Misc

Anyone who has seen their web server logs, has seen the Yahoo! crawler come by and leave its mark. It identifies itself as ‘Yahoo! Slurp’ and its domain is inktomisearch.com. For those of you who follow the search space, you can easily guess that this is one of the last historic remnants from our acquisition of Inktomi a few years ago. Well, the crawler finally decided to move and find a new home. We are moving our crawler from inktomisearch.com to crawl.yahoo.net.

The user-agent will continue to be Yahoo! Slurp, hence you do not need to make any changes to your robots.txt file. Nor are we changing the actual IP addresses of the crawler infrastructure during this transition. However, if you do any reverse DNS checks for the crawler identity or have any network access rules to allow inktomisearch.com, please also update them to allow for crawl.yahoo.net.

The crawlers will be switched over in phases starting immediately. You’ll see an increasing number of Yahoo! Slurp visits from crawl.yahoo.net and fewer and fewer from inktomisearch.com during the transition, which will take a few weeks.

Keep an eye on this space and the Yahoo! Search Blog for more details on when the migration is complete and details on bot verification with the new domain name.

Priyank Garg
Product Manager, Yahoo! Search

  • 21 Comments
  • Subscribe

RSS feed

21 Comments

Comment by Mark Barrera
2007-03-28 08:09:16

Thanks for the update!

 
Comment by Elmer W. Cagape
2007-03-28 18:18:19

Thanks Priyank, are there significant changes apart from the change in domain name?

 
Comment by domain inform
2007-03-28 19:56:53

great stuff I glad I found out that! good luck with the change yahoo!

 
Comment by Amit Doda
2007-03-28 22:00:14

Thanks for the information.. It’s very useful.

 
Comment by Autoverzekering
2007-03-28 22:48:06

Thanks a lot for the information.

 
Comment by mark
2007-03-29 01:43:37

thanks for the update, i have blogged it in.

 
Comment by simleon
2007-03-29 01:56:04

Hi, good to know.

Is it possbile to know why this change? Something ralated with new crawling system?

 
2007-03-29 04:03:26

Thanks for information. Hope that information current improvement or change will be available continue.

This is OK that “user-agent will continue to be Yahoo! Slurp”. Otherwise lots of seo have to change their website’s robot.txt file.

 
Comment by Nirupam Roy
2007-03-29 22:24:19

Thanks Priyank

Thanks for the info.

 
Comment by Peter Davis
2007-03-30 19:09:05

Does that mean you’ll be fixing Slurp so it won’t attack my site like a DOS attack anymore? Currently, I have hundreds of Slurps attacking my servers. Pretty much business as usual.

I like my site being indexed, but certainly you can do better. Google does it with just a few bots at a time. And, Google does a far, far, faaaaar better job of indexing the site with less resources.

 
Comment by Tilak
2007-03-30 22:23:04

Thanks for heads up!!!

 
Comment by gutschein
2007-04-08 06:04:57

I searched a long time for such an great article. Thank you

 
Comment by Julie
2007-04-12 16:07:59

Thank for the Info !

 
Comment by massive penis
2007-04-25 05:56:49

Yes truly great article

 
Comment by Home Furnishings
2007-05-21 11:49:24

Thanks for the info!

 
Comment by Scented Soy Candles
2007-05-21 11:51:39

Thanks sir!

 
2007-05-21 11:57:45

This is a very useful info for webmaster. Thanks.

 
2007-05-21 11:59:02

God bless yahoo!

 
Comment by Adnetwork Reviews
2007-06-07 08:43:30

Thanks, it’s great to see this change, now I can be sure those bots are yours. Will the bots now be more efficient in crawling sites?

 
Comment by Justin
2007-07-03 10:50:14

I’m not sure about anyone else, but I’ve seen the bots crawling more often lately. Lets keep it that way.

 
Comment by Nadir
2007-07-05 10:44:24

what’s the user agent of your mobile crawler please?

 

Sorry, the comment form is closed at this time.

back to yahoo! search

subscription options

Facebook Fans

latest posts

archives