« Who Doesn't Love Samples? | Main | Weather Report -- Major Crawl Improvements »

August 21, 2007

Be Dynamic, Be Confident -- Yahoo! Search Supports You

Please excuse the dramatic start to this post. Between the anticipation of rolling this out and my incessant Harry Potter reading, I couldn't resist.

Once upon a time, on the World Wide Web, all URLs were fixed strings -- static in form. The idea of URL parameters then came along, allowing for database driven sites and session ids in URLs to create personalized experiences for users. At that time, the Web was alive with rich data and experiences. Then came the crawlers, which made it easier for users to navigate through the Web; however, they inevitably battled with dynamic URL parameters and every webmaster had to choose between a dynamic site and search traffic.

Today comes a new wave for search engines with the first-ever Beta launch of 'Dynamic URL Rewriting' in Site Explorer. The new feature provides the ability for site owners to alert Yahoo! of the dynamic parameters in URLs that they'd like Yahoo! to ignore, which we'll then automatically rewrite accordingly. Try this out for all the cases where you'd want to use parameters in your URLs that don't affect the content of your page, but that have other important uses.

How to get there?

  • Login to Site Explorer from Yahoo! Search.
  • Add to My Sites and then authenticate any sites that you own or manage.
  • For any sites that you have authenticated, you'll see a 'Dynamic URLs' tab.
  • On this tab you can enter parameters you want us to either remove from URLs or always crawl with a specific value.
  • Once you enter the parameter, we'll show you the # of URLs we estimate will be affected.
  • After you confirm the action, we'll modify our crawler such that every time we see a URL from your site with that parameter, we'll automatically rewrite it within our system as per your instruction.

So you might wonder what the feature really gives you. Utilizing the 'Dynamic URL Rewriting' feature enables:

  • A more efficient crawl of your site, with fewer duplicate URLs being crawled.
  • Better and deeper site coverage, as we'll be able to use our crawler capacity to find and index more new content on your site.
  • More unique content discovered, as we'll handle more dynamic parameters in your URLs (if you remove the content-neutral dynamic parameters).
  • Fewer chances of crawler traps, or web page sets that can cause an infinite number of requests or a poorly constructed crawler to crash.
  • Cleaner and easier-to-read URLs displayed in the search results.
  • Better site ranking due to reduced fragmentation of links and anchor text to your site's pages.

Looking for more details on when to use URL parameters? Visit the Site Explorer Help page for additional background on the Beta feature to help define and omit what dynamic URLs Yahoo! should ignore.

We're here to address any questions/ needs that you have, so let us know how it works for you.


Priyank Garg
for Lakis, Amit B., Amit S., Jay, Judy, Srikanth, Zheng
Yahoo! Search

Permalink | Bookmark | del.icio.us | digg | Filed In: Site Explorer

Comments

Excellent! You guys keep catering to the people that write most of the content on the internet and you may just see a ground swell of endorsements and increase in market share.

Take that 100 million dollars that Ask wasted on advertising and put it into development into tools and interfaces for webmasters and you will have a huge machine pushing your services.

Google has a giant chink in their armor, complete lack of communication with the community that creates its content, you take advantage of that and in the long term you'll win.

Posted by: JLH | August 21, 2007 08:54 AM

So, would that essentially perform the same function as doing a server-side 301 redirect?

Posted by: Tim | August 21, 2007 09:19 AM

Or is this better than a 301?

Posted by: Tim | August 21, 2007 09:25 AM

If you use 301 redirects on the server side targeted at our crawler, you will have avoided the duplicate crawling and indexing of content.

However, the web references to pages on your site can still be fragmented due to such parameters being present on them. This feature addresses that as well by making all those links refer to the real pages after rewrites (instead of different URLs for the same page).

Posted by: Priyank Garg | August 21, 2007 10:05 AM

We have seen from our research that users do read URLs to establish context of which site they are going to. There is other 3rd party research that supports that. Clean simple to read URLs can make a difference to the visibility of your search results

Posted by: Priyank Garg | August 21, 2007 11:54 AM

This is very useful new feature we all can use.

Posted by: Decorative Throw Pillows | August 21, 2007 12:26 PM

This is really a cool feature. I didn't think we'd have any stray variables lying around but sure enough, there's a referral id that was being found! It's great that you give out the info on which parameters you suggest as well as the info on how many pages it will roughly effect if you type in your own parameter. This offers some very interesting data for analysis as well :)

I'm impressed and that's not just to do with this. Your search results have less spam than G's 8 out of 10 times...

Posted by: Sam Daams | August 21, 2007 02:52 PM

Nice useful feature. Great job guys!

Posted by: Vladimir | August 22, 2007 02:37 AM

This is really excellent, I have had some clients that have been wanting support for this.

Posted by: Ben P | August 22, 2007 09:32 AM

Forgive me for being dense, but I'm logged in and I'm seeing my authenticated sites, but no where can I find the "Dynamic URLs" tab.

Posted by: mphung | August 22, 2007 10:53 AM

wow......this is a great new. I wonder if Google is going to follow you guys on this. Excellent job!

Posted by: Chan Mom | August 22, 2007 01:06 PM

@Ben P: you need to click the [Manage] button for one of your authenticated sites and then you'll get to the page where you can see the [Dynamic URLs Beta] tab.

Posted by: Tim | August 22, 2007 02:24 PM

What most people are asking now is why their website do not appear in Yahoo,when they rank top 10 in google. What' so different in yahoo to do that. I can see those post "why I am not in yahoo" on every forum.

Posted by: Dmitri | August 22, 2007 02:44 PM

Hi Dmitri, you must have reported Spam to Yahoo.

Posted by: Anon | August 22, 2007 06:19 PM

this tool is really cool! is there also a way to do this at googles?!?

Posted by: piq | August 23, 2007 12:07 AM

Thanks for accepting dynamic urls.

Posted by: Tech For Novices | August 23, 2007 01:07 AM

Priyank Garg - are you saying that a 301 redirect would not pass on all the 'link juice' then?

Posted by: Ian | August 23, 2007 02:36 AM

A 301 redirect would pass all the link juice if the original URL form is crawled and as a result the new URL form is 301 redirected to. However, if you have the same page but different URLs with different session ids referred to across the web, the link juice of each of those URL references would not transfer to the final form till we finished crawling each of the original references.

With this tool, we would transfer the link juice the moment we see the form with session ids that are rewritten show up on the web without waiting for those URLs to be crawled. In some cases, we may choose not to crawl some of the URLs anyways due to crawl load and scheduling and this prevents that link juice being lost.

Posted by: Priyank Garg | August 23, 2007 05:47 AM

This new tool seems to be cool but we shall see.

Posted by: The Dog Clothing Company | August 23, 2007 07:25 AM

Great!

You really need to catch Google...

Keep up the good work

Posted by: Sourav | August 23, 2007 09:26 AM

status: Invalid ... Parameter too rare to validate the effect of this action !?!

this shows up in all my remove from url actions!

what does that mean? the site is authenticated ...

Posted by: piq | August 24, 2007 06:41 AM

THANK YOU, this is VERY helpful! The interface to manage the parameters is easy and FAST too--nice work!

MC

Posted by: Michael Cottam | August 24, 2007 08:01 AM

We offer thousands of products at below wholesale prices. Check us out today and save! Look at some of the things we have. Apple Ipods New And Refurb,Car Audio, Cd Players,Cellular Phones,Computers & Peripherals, Digital Cameras, Dj Equipment, Dvd Players, Fax Machines, Gps/cb Radio, Home Stereo Systems, Home Theaters, Televisions, Kids Electronics and alot more. Check us out at www.kforcestores.com and Save! We can set you up with your own online store like this. Anyone can make a sale on eBay, but not everyone can make a business out of it. Attention eBay Power Sellers! We can help you seek the perfect products at the lowest prices possible, for the greatest potential profits.
www.kforcestores.com

Posted by: Kory | August 25, 2007 12:12 AM

Hey yahoo,

It's an excellent news. Now we don't have to be worried about the way the dynamic URL is shown over the search engine. we can be sure that you will remove the dynamic characters...

Thanks Pal!

Posted by: Preeti | August 27, 2007 12:58 AM

Is this just for newly added sites? I don't see the tab for sites I currently have authenticated with Site Explorer.

Posted by: esoos | August 27, 2007 11:13 AM

I dont know how to proceed in the following matter, can you help
me?

I have my website authenticated in two diferent accounts, and in the first one, I already submit our sitemap. Right now, I am looking to move
it from the first account to this one (diretoriazura), How do I proceed?

Can I submit two diferent sources for the sitemap? Or I just have to move the feed to this one?

Best,
Marcelo

Posted by: Marcelo | August 27, 2007 12:19 PM

thats good, i will try to do this

Posted by: custom tailor thailand | August 27, 2007 08:02 PM

Does the yahoo links badge work? I just installed it & the badge is there but it doesn't show the number of links to my site. Does it take some time before it does show? Help.

Here's where I installed it;
http://olympicblogger.blogspot.com

Posted by: CallCenterVet | September 1, 2007 11:26 PM

Great idea. Am away to go and get all those horrid session ids etc off of some sites today - I'm useless in dot net so this should make this a bit simpler - if only everything could be on linus servers.

Posted by: Mike | September 3, 2007 01:09 AM

Thanks for this great and useful new feature. I definately looking forward to use it.

Posted by: magnetic jewelry clasp | September 5, 2007 06:56 AM

Hi..I don't see the dynamic URLs tab. Should I be trying something else?

Posted by: Josh | September 7, 2007 11:53 AM

Site Explorer is awesome! It's great to see you are trying to make this tool even better. What other cool things do you have planned?

Posted by: Duct Cleaning Guy | September 9, 2007 02:31 PM

Wow! No doubt that you're on No.2 position after Google in online terms. Hoping for more new updates.

Posted by: Web Developer India | September 12, 2007 03:48 AM

is this effect other search engines also or only yahoo

Posted by: Seo Expert | September 15, 2007 10:31 AM

We are finally starting to realize how crucial yahoo is to our business.

Posted by: women in stockings | September 18, 2007 11:46 PM

Seems like a nice bone to toss. Although companies should be addressing this at their end, ideally (by doing their own rewriting). Actually, the point of my post was to promote decorative throw pillows, as well as to say nice job!

Posted by: Andrew Goodman | September 26, 2007 06:59 PM

I think that a well structured web site with the right code (asp.net or php or everything you want) could certainly avoid the problems of the URL rewriting doing it internally and avoiding the necessity to use external resource like yours.

You have done probably a great job, but can a single search engine be the specific recipe to cure all the worst? I think no, and this is the reason IMHO for which a lot of webmaster won't use this.

Bye

Posted by: Indicizzazione motori Web-MA | September 27, 2007 03:56 AM

Yahoo Dynamic Url Rewriting and yahoo pipes are the most original services powered by Yahoo!

Innovate and dominate !!!

Posted by: Seo Toolbar | September 28, 2007 05:59 PM

I added my site. Thanks Yahoo!

Posted by: JavaScript tools and tutorials. | September 29, 2007 08:09 PM

Its really nice...
I want to know, is this for yahoo only or for all search engines...

Posted by: Integrated Product Development Group | October 3, 2007 12:59 PM

Yahoo Rocks! We love the new search and the site explorer. Keep up the good work.

Posted by: Hughes Commercial Businesses for sale | October 4, 2007 05:00 AM

I appreciate that you keep your site going and allow comments. To me feedback is an interesting part of a website.

Posted by: Melissa | October 16, 2007 11:00 PM

I want to know, is this for yahoo only or for all search engines...

Posted by: kale kapi | December 17, 2007 01:25 PM

appreciate that you keep your site going and allow comments. To me feedback is an interesting part of a website

Posted by: ev | December 17, 2007 01:26 PM

I appreciate that you keep your site going and allow comments. To me feedback is an interesting part of a website.

Posted by: best videos | December 23, 2007 08:12 AM

Yahoo rocks. I get most of my web services from yahoo.com

Posted by: Tom Muse | January 4, 2008 12:09 AM

Thanks. This is a great feature that I've been looking for a mainly database website, hosted in a Windows environment.

Posted by: SEO Consultant | January 14, 2008 09:39 PM

Yahoo Dynamic Url Rewriting and yahoo pipes are the most original services powered by Yahoo! thanks yahoo :)

Posted by: ensest hikayeler | January 15, 2008 08:19 AM

i havnt see the aboive mentioned "dynamic url"..in my siteexplorere page.?
How i got it?

Posted by: priyank | January 27, 2008 11:50 PM

A little off topic, but I have always avoided Flash graphics as search engines including Yahoo aren't very good at reading them. Is there any way in the future that Yahoo could read Flash as they could normal test and include this in their crawling?

Posted by: Neil Simmons | January 28, 2008 06:56 AM

This is definitely a move in the right direction.

Posted by: Matt Harrison | January 28, 2008 07:01 AM

Dynamic URL rewriting is a good thing, and so is Harry Potter!

Posted by: Bob Simpson | January 28, 2008 07:04 AM

i havnt see the aboive mentioned "dynamic url"..in my siteexplorere page.?

Posted by: kale servis | January 28, 2008 07:25 AM

I still don't understand why are Dynamic URL's are so beneficial.

Posted by: Property bangkok | January 30, 2008 04:57 AM