Weather Report — Major Crawl Improvements
- Posted August 22nd, 2007 at 8:00 am by Yahoo! Search
- Categories: Weather Report
If you haven’t been busy at SES in our neck of the woods this week, you might have noticed we’ve been rolling out fresh updates to our crawl, index and ranking systems — ones that are relatively significant for webmasters.
In response to feedback from the community, we’re reducing our crawling machines by a significant amount with this update, so you might have observed a significant reduction in crawl load from us along with fewer machine IP addresses hitting you over the last few days.
The new crawler continues to improve comprehensiveness and freshness of coverage while also enhancing crawl efficiency, which reduces spurious load on websites.
We had a brief bout of increased crawl load while testing and rolling out updates, like support for ‘Dynamic URL Rewriting’ in Site Explorer, announced yesterday. Don’t fret, though, if you’re concerned about seeing an increased load. We’ve initiated efforts and established policies internally to ensure this doesn’t happen, even temporarily, in the future.
Thanks and keep the feedback coming.
Priyank Garg
Yahoo! Search
- 34 Comments
- Subscribe
Here is one for the record books for people who want to report spam. Two days ago we were at the ends of our rope with a competitors spam. So
we decided to send of an email detailing the spam, what happed was we
were reprimanded, I guess for reporting it we know can be found on page 22.
Advise to others who want to report SPAM…Don’t Do It.
Hi Priyank
Will this improve or worsen searching capabilities of Yahoo for blogspot domains?
Good news guys. I hope more people catch on to the quality of your index and searchers switch over more. Searches for things like ‘free ringtones galore’ or ‘travel blogs’ has you guys bringing up much better results than Google.
Hello,
I just tried the new Yahoo beta last night, and it does not load, just gets stuck. I ws trying to get it to work for 3 hours yesterday and again this morning. Pls let me know how I can get my old Yahoo mail back, because I deparately need to access my emails and I cannot!
Thanks for the help!
Great move! Yahoo crawler finally discovered many of the subpages on my sites (from only a few to a few dozens overnight). I also saw a dramatic improvement on my ranking. Good for me!
It is sad to say, but I see alot of subdomain spam in my industry. I dont follow other ones, but mine is flooded.
Can anyone say if the update is complete?
We have great new rankings from where I search, but clicks haven’t gone up much, so I’m fairly sure it hasn’t rolled out to all datacenters yet…
I’m going to pay close attention to what impact the update of this time give us.
My website ranking has been plummeted drastically in these updates. Our site was no.1 and now its on 15th. How should I suppose to pay my employees if my website drops without no reason whereas its working fine in other SE’s.
What I immediately noticed was a lot less load on my webserver. My server runs 70+ websites and slurp was killing my server. My average server load has gone from 4.0+ to < 0.7. Good Job Yahoo Crawlers!
Secondly I have noticed Yahoo Search returns more results for sub-pages on my sites then it ever has in the past. It tended to be home page centric and that is still happening but with the bonus of finding nuggets of good results on pages 2,3,4 deep off the home page. Nice job Y! Search!
yahoo/inktomi crawling is still not that good… i just installed and deployed Yahoo tags for tracking and conversion… and yahoo crawlers hit us so aggressively during business hours that it crashed our servers… we had to block all yahoo servers to fix this… you would think that they would only crawl at reasonable off hours as google does, which i never had an issue with
It appears to be an improvement. Now all you have to do to get back your search market share is:
1) Remove all of the old Yahoo Directory listings from Yahoo Search. You’re still showing the HTML Title to our front page that we submitted to the Yahoo Directory 5 years ago. I’ve heard this same complaint from numerous other people.
2) Yahoo still has the reputation as the most policed and Draconian of all search engines. Yahoo bans more legitimate sites(only indexes their front page) than all of the other engines combined. Is this editorial policing good? Look at your market share. You’ll soon be below 20%.
SFWriter777: Two options: update your Y! Directory listing or use NOYDIR as described here: http://help.yahoo.com/l/us/yahoo/search/indexing/indexing-11.html
Tim:
Thank you for the suggestion. However, I have tried to contact Yahoo Directory to change the HTML Title to what we’ve had for the past 5 years. I received a form email basically saying that “It didn’t meet their guidelines.”
And that is precisely my point. Nevermind that Google and MSN have had no problem with our title for years (we are currently ranked # 2 and # 3 respectively on those sites). The bottom line is that Yahoo’s editors are out of control. They have scrutinized and policied Yahoo to the point where people are not finding what they want and are simply looking elsewhere. It will only be a matter of time before Yahoo slips below 20% market share.
Case in point: Dallas.com (not my site). Great site, extremely popular and does a service to those looking for information about Dallas. Does well on Google (# 7) and MSN (# 1) for a search of Dallas. It is not even in the Top 50 in Yahoo Search. I have no idea and I’m sure they don’t either. The problem is that Yahoo has long penalized decent sites for technical reasons that no longer make sense to anyone but themselves.
My industry still has alot of spam and old URLs. I see alot better results in other industries maybe it just harder to contain people’s tactics my industry.
If you can cut down on spam url results all the better!!
I noticed less trafic on servers to of Slurp, Yay. It has affected my rankings to however. Dropped from third to eleventh. Is it not becoming to strict? Thanks for the efforts in reducing spam.
If you can cut down on spam url results all the better!!
My industry still has alot of spam and old URLs. I see alot better results in other industries maybe it just harder to contain people’s tactics my industry.
good ;)
“I love yahoo search, less spam and better search results.”
i think too as history
love yahoo search, less spam and better search results.”
i think too as history
I’m going to pay close attention to what impact the update of this time give us.
It is sad to say, but I see alot of subdomain spam in my industry. I dont follow other ones, but mine is flooded.
My website ranking has been plummeted drastically in these updates.
We have great new rankings from where I search, but clicks haven’t gone up much, so I’m fairly sure it hasn’t rolled out to all datacenters yet…
We have great new rankings from where I search, but clicks haven’t gone up much, so I’m fairly sure it hasn’t rolled out to all datacenters yet…
I’m fairly sure it hasn’t rolled out to all datacenters yet…
We have great new rankings from where I search, but clicks haven’t gone up much, so I’m fairly sure it hasn’t rolled out to all datacenters yet.
Seeing an increased load is fine, at lease I know that Yahoo is being active.
I have for a while had very high position on Yahoo for my key search terms. So for me, Yahoo is picking up on what I do and what I want other people to know that I do.
I think Yahoo’s algorithm and search system is great, it’s just a shame that Yahoo itself doesn’t yet more of the global traffic.
I always get more hits from inktomi slurp than any other search engine spider
Is the number of hits directly proportional to search engine placement?
I get the same as peter, more hits from inktomi.