« It?s Search. It?s Site Explorer. It?s Webzari! | Main | And The Answer Is... »
Weather Report ? The New Yahoo! Search Crawler (Slurp) Is Here!
We launched a new Yahoo! Search Crawler, Yahoo! Slurp earlier this week. In addition to crawling the Internet faster, our new crawler is more efficient at visiting websites. As a result, site owners will notice as much as a 25% reduction in the number of requests and bandwidth consumed by the crawler.
While transitioning to the new crawler over the past few weeks, we had been running both crawlers in tandem. In some cases, this increased the frequency of Yahoo Search requests to websites. Now, with the new crawler in full production, we have turned off the old crawler and site owners should see a much lower crawl load without a loss in content coverage.
With this change of behavior in the crawler, you may see some shuffling of the pages that are included in the index and some changes in ranking as well.
Let us know what you observe through our feedback form, or if you have any technical issues with Slurp please contact support.
Thank you to everyone who helped us with this update!
David Simpson
Yahoo! Slurp Team
Priyank Garg
Yahoo! Search


Comments
Thank you for making this change! I can already see a lower crawl load from Slurp, with minimal changes in the rankings for the sites that I watch.
Brian M
Posted by: Brian M | July 28, 2006 08:55 AM
I am seeing to many directories and .gov and .org, and russians sites in the results. Right now its looks like a bad copy of Google it was much better before.
Posted by: gary | July 28, 2006 09:43 AM
I thought I'd noticed a reduction in the number of hits from Slurp recently! Glad to see it wasn't Slurp deciding it didn't like my site. :)
Posted by: Alden Bates | July 28, 2006 03:22 PM
The old results were better. The new cralwer has brought in lot of trash results for atleast the industries that I follow.
Posted by: Chris | July 28, 2006 04:12 PM
Nice!
Posted by: Joerg Battermann | July 29, 2006 01:59 AM
Tats really good. We need to thank yahoo for making this change. Hope the new slurp is healthy and eats only fresh food and generates better indexes and SERPs thereforth.
Posted by: Thomas | July 29, 2006 04:05 AM
Good News for Web Owners.
Posted by: Jim | July 29, 2006 09:36 AM
Thank Goodness! Slurp has really been humping away at my sites over the past few months.
Posted by: Peter | July 29, 2006 10:10 AM
Thanks for the updated crawler, as any update is appreciated by those of us in this community. I just hope some of my links come back!
Cheers.
Posted by: Rob | July 29, 2006 06:39 PM
Hi, Interesting changes ! Just one observation regarding my search term "Edinburgh cricket"; why does www.123-reg.co.uk come 2nd !? It's a domain registration page - not a cricket site. I'm sure it's just a glitch which will be ironed out soon. Cheers, AL.
Posted by: AL | July 31, 2006 12:53 AM
Hi,
One thing i would like to see is date when last cached information.
When i see my website in yahoo search results and click on the 'cached copy' i don't see the date this cached data is from.
Is it possible to get this information through other means?
My website changed from second position to third recently and hope that this is the result of new crawler.
--sri
Posted by: Sridhar | July 31, 2006 07:26 AM
A step in the right direction. Nice to see the search results updating much more frequently.
Posted by: Squiggs | July 31, 2006 08:36 AM
I'm already noticing an improvement in the indexing (i.e. not indexing pages that we give you a 404 on).
One remaining indexing issue that I see is the main root page is getting listed second on a site: command, so we see something like this on a site:domain.com command
1. www.domain.com/somerandompage
2. domain.com
3. www.domain.com/someotherrandompage
Any reason why the root would show second, and without www?
Joe
Posted by: Joe | July 31, 2006 10:28 AM
Will this new slurp stop covering error logs with SlurpConfirm404 's ?
Posted by: Thomas Tallyce | July 31, 2006 10:47 AM
How many times does the bot HAVE to reqest the robots.txt file in a day? Seems to do it like every few page requests.
Posted by: Arubicus | July 31, 2006 11:50 AM
Does your new crawler read Javascript?
Benoit
Posted by: Benoit | July 31, 2006 11:55 AM
Well, new or old, it doesn't really matter. The fact is that there are a lot of glitches with Yahoo's search result. I am not gona vent or something but I believe that Yahoo has more spam sites than ever. Only because a dude has all the time in the world to get inbound links, he doesn't deserve top rankings. I am not happy at all, specially with Yahoo's ageing issues! Not a happy camper I must say
Posted by: Zeeshan Syed | July 31, 2006 03:24 PM
not happy with yahoo results. There is lot of spam and free sites coming in top. Sites coming in top rankings does not have good Backward links nor content, so why such sites are coming on top.
The aim of launching the new crawler might be anything but you should show good results than Google.
Posted by: Clara | July 31, 2006 10:01 PM
Ever since the new slurp came out we see it spider our site but no updates to the pages and the pages are slowly disappearing from yahoo.
Seems like slurp hits the robots file an awful lot and then some pages it just keeps hitting a few times a day, but no updated cache's on yahoo and the pages are slowly disappearing.
The site is a clean site that never had any problems on yahoo before, I don't know what is going on.
Posted by: RKR | August 1, 2006 05:04 AM
Data set looks ok. not quite as good as before however the hits to servers look like they have decreased. I guess that the next update to the algo will repair any gaps in the logic.
Thanks for the Update
Posted by: Mike | August 1, 2006 06:25 AM
The robots are reading the "disallow: /file/" and a "site:mysite.com" shows all the old pages are gone, a step forward.
The Directory Title and Descriptions are still overtaking the site meta tag, this problem needs a permanent fix.
As a result of the update, we have lost about 40% of Yahoo traffic, do to Spammers, Geocties, Tripod and the like.
Baby Steps keep it up.
Blog
Posted by: Blog | August 1, 2006 10:26 AM
Whatz wrong with Yahoo result.It seems every active site, they lost postion in yahoo search, I only find dead sites in first results.
Posted by: john peterson | August 1, 2006 12:44 PM
I notice that our site is dropping out of the search engine now. It is a clean site addressing skin condition and used to be on the first page of yahoo search results, what happened??
Posted by: jo | August 1, 2006 10:26 PM
No Yahoo - this is not on - lost lots of rankings on my own as well as client sites - all of a sudden. Stop trying to be like Google - just be yourselves guys
Posted by: Deon Coetzee | August 1, 2006 11:23 PM
Thanks for the Weather Report.
One oddity I've noticed since July 28 is as follows: For a certain two word search with about 9 million results, there is one index that returns MySite as #1 - which it was consistently for several months prior to July 28. Another index has MySite 'missing' for this same search (by missing I mean not in top 100 results). For a different two word search, MySite is still first page on the later index, but dropped to fourth page on the former. Yet both indices seem to return MySite for a variety of other searches about the same as prior to July 28 - at least not any substantive differences I've noticed. I recall in 2004 and maybe early 2005 noting that Yahoo seemed to have 3 distinct indices, but in 2006 I don't recall seeing such violent differences in search results. I don't know if it is cookies, IP address or phase of the moon that causes the different indices to be be used. Are several indices in the process of being merged or is this some kind of A/B testing? Danke (Go ahead and tell us - we'll keep your answer a secret.)
Posted by: Confused | August 2, 2006 11:12 AM
OMG - this has trashed some of my clients listings. What the hec did you do? There's a load of trashy results in place of the relevant listings that existed before hand. The results are no longer to be seen in the index. Nowhere at all. Something has majorly gone wrong. Put it back!!!
Posted by: TomD | August 3, 2006 09:51 AM
Great I lost all my rankings again. First the weather update now the new crawler. It seems like everytime you guys try to do something new, we see more spam in the serps. Keep up the good work.
Posted by: Mikey | August 3, 2006 09:56 PM
Welp our site has lost tons of pages now -- Go figure. Oh well what can you expect.
Posted by: Arubicus | August 4, 2006 10:21 AM
I really hope someone can help here. It seems like the new bot just hits the same pages over and over and we are losing pages in the yahoo index. Then I discovered that when I log into yahoo site explorer it says my site has one subdomain, we have never had a subdomain. Could this all be connected somehow?
Posted by: Robin | August 11, 2006 08:41 AM
In the last month my site has virtually disappeared from the yahoo search index. I had about 250 pages indexed, now I have 11. I went from 500 unique impressions a day to ZERO. My site is totally clean with no tricky linking whatsoever, no subdomains. Every page has over 350 words of original content. I had dozens of page 1 entries for a wide variety of keywords, now I have ZERO. Two years worth of work with good steady revenue has been reduced to a big GOOSE EGG. What a bummer!!!
Posted by: Chuck | September 23, 2006 03:10 AM
What happened to the Slurp ? It only indexes 1/4 of what it used to index before. Also, may High PR inbound links are not getting indexed.......
Posted by: Badminton Blog | October 4, 2006 01:27 PM
thanks!
Posted by: msn nick | April 6, 2007 01:10 PM
I hope that all those changes will have good effects on the way Yahoo! works.
Posted by: SEO company: White Hat Media | April 17, 2007 06:33 AM