Weather Report ? The New Yahoo! Search Crawler (Slurp) Is Here!
We launched a new Yahoo! Search Crawler, Yahoo! Slurp earlier this week. In addition to crawling the Internet faster, our new crawler is more efficient at visiting websites. As a result, site owners will notice as much as a 25% reduction in the number of requests and bandwidth consumed by the crawler.
While transitioning to the new crawler over the past few weeks, we had been running both crawlers in tandem. In some cases, this increased the frequency of Yahoo Search requests to websites. Now, with the new crawler in full production, we have turned off the old crawler and site owners should see a much lower crawl load without a loss in content coverage.
With this change of behavior in the crawler, you may see some shuffling of the pages that are included in the index and some changes in ranking as well.
Let us know what you observe through our feedback form, or if you have any technical issues with Slurp please contact support.
Thank you to everyone who helped us with this update!
David Simpson
Yahoo! Slurp Team
Priyank Garg
Yahoo! Search
Thank you for making this change! I can already see a lower crawl load from Slurp, with minimal changes in the rankings for the sites that I watch.
Brian M
I am seeing to many directories and .gov and .org, and russians sites in the results. Right now its looks like a bad copy of Google it was much better before.
I thought I’d noticed a reduction in the number of hits from Slurp recently! Glad to see it wasn’t Slurp deciding it didn’t like my site. :)
The old results were better. The new cralwer has brought in lot of trash results for atleast the industries that I follow.
Nice!
Tats really good. We need to thank yahoo for making this change. Hope the new slurp is healthy and eats only fresh food and generates better indexes and SERPs thereforth.
Good News for Web Owners.
Thank Goodness! Slurp has really been humping away at my sites over the past few months.
Thanks for the updated crawler, as any update is appreciated by those of us in this community. I just hope some of my links come back!
Cheers.
Hi, Interesting changes ! Just one observation regarding my search term “Edinburgh cricket”; why does http://www.123-reg.co.uk come 2nd !? It’s a domain registration page – not a cricket site. I’m sure it’s just a glitch which will be ironed out soon. Cheers, AL.
Hi,
One thing i would like to see is date when last cached information.
When i see my website in yahoo search results and click on the ‘cached copy’ i don’t see the date this cached data is from.
Is it possible to get this information through other means?
My website changed from second position to third recently and hope that this is the result of new crawler.
–sri
A step in the right direction. Nice to see the search results updating much more frequently.
I’m already noticing an improvement in the indexing (i.e. not indexing pages that we give you a 404 on).
One remaining indexing issue that I see is the main root page is getting listed second on a site: command, so we see something like this on a site:domain.com command
1. http://www.domain.com/somerandompage
2. domain.com
3. http://www.domain.com/someotherrandompage
Any reason why the root would show second, and without www?
Joe
Will this new slurp stop covering error logs with SlurpConfirm404 ’s ?
How many times does the bot HAVE to reqest the robots.txt file in a day? Seems to do it like every few page requests.
Does your new crawler read Javascript?
Benoit
Well, new or old, it doesn’t really matter. The fact is that there are a lot of glitches with Yahoo’s search result. I am not gona vent or something but I believe that Yahoo has more spam sites than ever. Only because a dude has all the time in the world to get inbound links, he doesn’t deserve top rankings. I am not happy at all, specially with Yahoo’s ageing issues! Not a happy camper I must say
not happy with yahoo results. There is lot of spam and free sites coming in top. Sites coming in top rankings does not have good Backward links nor content, so why such sites are coming on top.
The aim of launching the new crawler might be anything but you should show good results than Google.
Ever since the new slurp came out we see it spider our site but no updates to the pages and the pages are slowly disappearing from yahoo.
Seems like slurp hits the robots file an awful lot and then some pages it just keeps hitting a few times a day, but no updated cache’s on yahoo and the pages are slowly disappearing.
The site is a clean site that never had any problems on yahoo before, I don’t know what is going on.
Data set looks ok. not quite as good as before however the hits to servers look like they have decreased. I guess that the next update to the algo will repair any gaps in the logic.
Thanks for the Update
The robots are reading the “disallow: /file/” and a “site:mysite.com” shows all the old pages are gone, a step forward.
The Directory Title and Descriptions are still overtaking the site meta tag, this problem needs a permanent fix.
As a result of the update, we have lost about 40% of Yahoo traffic, do to Spammers, Geocties, Tripod and the like.
Baby Steps keep it up.
Blog
Whatz wrong with Yahoo result.It seems every active site, they lost postion in yahoo search, I only find dead sites in first results.
I notice that our site is dropping out of the search engine now. It is a clean site addressing skin condition and used to be on the first page of yahoo search results, what happened??
No Yahoo – this is not on – lost lots of rankings on my own as well as client sites – all of a sudden. Stop trying to be like Google – just be yourselves guys
Thanks for the Weather Report.
One oddity I’ve noticed since July 28 is as follows: For a certain two word search with about 9 million results, there is one index that returns MySite as #1 – which it was consistently for several months prior to July 28. Another index has MySite ‘missing’ for this same search (by missing I mean not in top 100 results). For a different two word search, MySite is still first page on the later index, but dropped to fourth page on the former. Yet both indices seem to return MySite for a variety of other searches about the same as prior to July 28 – at least not any substantive differences I’ve noticed. I recall in 2004 and maybe early 2005 noting that Yahoo seemed to have 3 distinct indices, but in 2006 I don’t recall seeing such violent differences in search results. I don’t know if it is cookies, IP address or phase of the moon that causes the different indices to be be used. Are several indices in the process of being merged or is this some kind of A/B testing? Danke (Go ahead and tell us – we’ll keep your answer a secret.)
OMG – this has trashed some of my clients listings. What the hec did you do? There’s a load of trashy results in place of the relevant listings that existed before hand. The results are no longer to be seen in the index. Nowhere at all. Something has majorly gone wrong. Put it back!!!
Great I lost all my rankings again. First the weather update now the new crawler. It seems like everytime you guys try to do something new, we see more spam in the serps. Keep up the good work.
Welp our site has lost tons of pages now — Go figure. Oh well what can you expect.
I really hope someone can help here. It seems like the new bot just hits the same pages over and over and we are losing pages in the yahoo index. Then I discovered that when I log into yahoo site explorer it says my site has one subdomain, we have never had a subdomain. Could this all be connected somehow?
In the last month my site has virtually disappeared from the yahoo search index. I had about 250 pages indexed, now I have 11. I went from 500 unique impressions a day to ZERO. My site is totally clean with no tricky linking whatsoever, no subdomains. Every page has over 350 words of original content. I had dozens of page 1 entries for a wide variety of keywords, now I have ZERO. Two years worth of work with good steady revenue has been reduced to a big GOOSE EGG. What a bummer!!!
What happened to the Slurp ? It only indexes 1/4 of what it used to index before. Also, may High PR inbound links are not getting indexed…….
thanks!
I hope that all those changes will have good effects on the way Yahoo! works.