These days, search has become such a fact of life, that everybody I talk to takes it completely for granted.

But it seems like every week I’m reminded why its so much fun working on a search engine that is used daily by millions of people, and most importantly by friends and neighbors and that lady in the store down the street. Take, for instance, last week …

I was out for coffee at the local Starbucks when the guy ahead of us in the line had lost the piece of paper with directions to the restaurant he was meeting friends at … and I was extremely happy to show him this:

So not only did it take me just one search to get the address and phone number, but now he has the listing and directions right on his cell phone … :)

What was that? It is a new feature in our abstract generating algorithm that tries to guess the most used information about that page, and promotes it into the summaries for search results. Not only do we try to figure out the most used information on the page, but we also integrate relevant features from other parts of Yahoo!. For instance, in the example above, you can see that we have found Maps & Reviews on Yahoo! Local, and also found that Yahoo! Local can send the address and phone number to your mobile phone – so thats there too! But of course, its not always just content from Yahoo! – we also use content from the site itself if that’s more relevant – try searching for Wal-Mart or 511 or FedEx …

Whaddyathink? Go ahead! Try it out. And perhaps next time you need directions for that restaurant, or suddenly crave some ice-cream (or coffee), or want a quick look to see if you won the lottery, and maybe (just maybe) see if the lottery money somehow made it into your bank account Yahoo! Search might have the answer right there for you.

Kalpana Ravinarayanan

Product Manager, Yahooo! Search

When we href="http://www.ysearchblog.com/archives/000074.html">first
released Y!Q Contextual
Search we also provided a href="http://yq.search.yahoo.com/publisher/examples.html">Y!Q
Demobar for download. With the Y!Q Demobar, you could highlight
not just a few words, but an entire sentence, a paragraph, or even a
complete article, and use that selection to trigger a search.

With the Y!Q Demobar, you could enjoy two of Y!Q’s most important
benefits:

1. Not having to worry about query articulation. Select as much
   text as you want and Y!Q will “automagically” determine the context
   from your selection and provide the most relevant results. (Try
   copying and pasting long text into a regular search box for comparison
   and you will notice the difference.)
2. Getting search results on the
   spot without leaving the page that you’re on. Y!Q gives you your
   results in a small overlay window. Some of my colleagues call this a
   “search snack.” (Does that make search results in a full-size window a
   search “meal”?)

We received lots of positive feedback after the initial release of
the Y!Q but many of you were asking: “Why do I need to download an
extra toolbar? Why not just add Y!Q to the regular href="http://toolbar.yahoo.com">Yahoo! Toolbar“?

Good news: We heard the feedback, and the latest version of the
Yahoo! Toolbar for Internet Explorer now supports Y!Q (integrated along with other cool
features, such as saving to href="http://myweb.search.yahoo.com/myweb">My Web and href="http://music.yahoo.com/">Y! Music Engine controls). Install
the latest Yahoo! Toolbar, select several words from the web page
you’re reading, and you’ll see an actuator icon. Select the “Search”
menu item and you’ll get contextual search results, right at “the
point of inspiration” when you see something interesting on the web
(see Figure below)

But we didn’t stop with Toolbar. We also added Y!Q to the new
version of Yahoo! Messenger with
Voice, within the href="http://messenger.yahoo.com/feat_search.php;_ylt=AtUgTHafZY.E6viMZSuVo_NnMMIF">LiveWords
feature. Activate LiveWords from the Yahoo! Messenger preferences menu
and interesting words in your IM conversation are under-lined. Click
on those words to trigger a search, or just as with Toolbar, select
message text to trigger a Y!Q search (see Figure below)

Moving towards a “select and click model” for searching takes a
little getting used to. I was used to sometimes typing in long and
specific queries in a toolbar or search box, which worked fine most of
the time. Now that I’m using the latest Yahoo! Toolbar and Yahoo!
Messenger, I don’t miss the typing or cutting-and-pasting at all.

We don’t think you’ll miss them either, so please give it a try
within Yahoo! Toolbar and Yahoo! Messenger, and let us know what you
think. We’re always interested in learning more on how to improve
Y!Q.

Reiner Kraft
Technical Yahoo!

One of the items I work on is href="http://tools.search.yahoo.com/shortcuts/">Yahoo! Shortcuts.
A Yahoo! Shortcut is a quick way to use web search to get to the
information you want, ideally faster than combing through the web
results. Shortcuts results appear automatically when they’re relevant
to your search and link to content on Yahoo! or across the web.
That’s a bit of an awkward description, so let’s get a better
definition and see a shortcut in action.

A web query for “ href="http://search.yahoo.com/search?p=define+shortcut">define
shortcut” gives The American Heritage Dictionary’s definition of
“shortcut” as “A more direct route than the customary one” in the
Yahoo! Shortcut section (look for the red “Y!” next to it near the top
of the web search results page. The pattern of “define x” can be used
to get definitions, as is pretty obvious, aka apparent, clear, evident
- you can get a whole list of synonyms by using “ href="http://search.yahoo.com/search?p=synonym+obvious">synonym
obvious“.

Back to the definition of “a more direct route”, there are a number
of patterns or key words you can use to get results from other areas
of Yahoo!. Looking at the tabs across the top of the search box:

• Images: “photos” or “pictures” in a web query will give 4 images
  and a link to all Yahoo! Image Search results. Example: “ href="http://search.yahoo.com/search?p=puppy+pictures">puppy
  pictures” (a favorite of my wife)
• Video: “video” or “videos” in the query gives a link to all Yahoo!
  Video Search results. Example: “ href="http://search.yahoo.com/search?p=brad+pitt+videos">brad pitt
  videos” (another favorite of my wife)
• Directory: some terms will trigger links to the Yahoo!
  Directory’s pages of categorized sites. Example: “ href="http://search.yahoo.com/search?p=history">History“
• Local: a pattern of [city] [state] [business name / type] will
  give 3 results from Yahoo! Local plus a link to all results. Example:
  “ href="http://search.yahoo.com/search?p=san+francisco+ca+plumbers">San
  Francisco ca plumbers“
• News: “news” in the query will gives three news headlines via
  Yahoo! News and a link to all results. Example: “ href="http://search.yahoo.com/search?p=Iraq+news">Iraq news“
• Products: various products or categories of products queries will
  give links to Yahoo! Shopping. Example: “ href="http://search.yahoo.com/search?p=digital+cameras">digital
  cameras“, “ href="http://search.yahoo.com/search?p=canon+powershot">canon
  powershot“

Many people also end up using the Shortcuts for navigation. For
example, “Pool” is
used by many to quickly get to Pool on Yahoo! Games. Some savvy users
just put an “!” on the end web queries that are Yahoo! properties to
be taken straight there. “ href="http://search.yahoo.com/search?p=mail!">Mail!” will take you
to Yahoo! Mail, “ href="http://search.yahoo.com/search?p=music!">music!” to Yahoo!
Music, “maps!”
goes to Yahoo! Maps, and so on. (Ok, you smart people will point out
that for maps you can just do a web search for an address and get a
map at the top of the page, such as “ href="http://search.yahoo.com/search?p=550+Geary+St%2C+San+Francisco%2C+CA">550
Geary St, San Francisco, CA“).

Lastly, with over 30 shortcuts on web search, there’s plenty of
ways to string them together to get things done faster. My wife will
be headed to Las Vegas with some friends next month (yes, it is a
bachelorette party and no, I’m not invited). She can use shortcuts to
book
her flight, look at prices and reviews for href="http://search.yahoo.com/search?p=las+vegas+hotels">hotels in the
area or a
specific one, and line up href="http://search.yahoo.com/search?p=limousines+las+vegas">a
limo. When it’s time to go, she can href="http://search.yahoo.com/search?p=las+vegas+weather">check the
weather, href="http://search.yahoo.com/search?p=san+francisco+airport">get
airport info, see if href="http://search.yahoo.com/search?p=united+1500">her flight is
on time, href="http://search.yahoo.com/search?p=san+francisco+traffic">check
traffic, and see href="http://search.yahoo.com/search?p=san+francisco+gas">where to get
cheap gas on the way to the airport.

Take a look at the href="http://tools.search.yahoo.com/shortcuts/">web search
shortcuts and see how they can help you find things faster. You
can always click on the Shortcuts link on href="http://search.yahoo.com/">search.yahoo.com, our dedicated search home page, or on the top right
of the web search results page, or by clicking the About link that’s
at the bottom of every shortcut. For access while on the go, take a
look at the href="http://mobile.yahoo.com/search/shortcuts">shortcuts for mobile
devices.

Adam Durfee

Product Manager, Yahoo! Search

My dad is a bit of a gardener and he’s also recently taken up
roasting tiny batches of coffee
beans. The results, besides earning the title of Papa Barista
from my sister in law, are some great coffee and tasty veggies. He’s
been tinkering on both fronts – in the short term, new types of coffee
beans every few weeks, and in the long term he’s changed the mix in
the garden, putting in more garlic this year, probably to tie in with
the bumper crops of basil he gets and his pesto making activities.

We’ve been doing a little tinkering of our own on the Search front
page at search.yahoo.com. No
surprise that the focus of the page is search, but we got requests to
add mail and news to the page, so we tried to do so with minimal
impact on those who weren’t interested. The result? We added
individual mail and news modules that you can show or hide at will, so
now if want to see your mail status or news headlines (or both),
you’ve got that option. If not, you can simply leave the modules
hidden. The choice is up to you.

Since the response has been good, we’ll continue tinkering. The
latest change to the page is the addition of a link to href="http://myweb.search.yahoo.com/myweb">My Web, our personal
search engine that allows you to save, recall, and share information
you find online (check out href="http://www.ysearchblog.com/archives/000104.html">Kevin’s
post for more info). In the short term we’ll continue to test
some different links and in the long term maybe change the mix a
little as to what is shown where.

What can I say? Like father, like son. Grab a cup of fresh coffee
(skip the pesto for now) and check the Search front page out at href="http://search.yahoo.com/" target="_blank">search.yahoo.com – we hope you
like it. If you think of something you would like to see on the page
or a change you think should be made, by all means, leave a comment
below.

Adam Durfee

Product Manager

Yahoo! Search

If you run a public webserver, you have likely seen our webcrawler, named Slurp, in your logs. Its job is to find, fetch, and archive all of the page content that is fed into the Yahoo! Search engine. We continuously improve our crawler to pick up new pages and changes of your sites, but the flip side is that our crawler will use up some of your bandwidth as we navigate your site. Here are a few features that Yahoo!’s crawler supports that you can use to help save bandwidth while ensuring that we get the latest content from your site:

Gzipped Files: Our crawler supports gzipped files to reduce bandwidth requirements. On average, you will get a 75% savings when you enable compression for your site. Many webservers provide mechanisms for sending out HTML content in a compressed format (for example, mod_gzip for Apache). How much of your site’s total bandwidth you can save will depend on how much of your content is compressed and how well it compresses. In general, static pages are good candidates for compression. Any user agent, whether it is a browser or a search engine spider, will let the webserver know it can process compressed content by adding “Accept-Encoding: gzip, x-gzip” to the header of its HTTP request. All major browsers support gzip compressed content. Also you should be happy to know that if our crawler has any trouble with a compressed page, it will re-fetch the uncompressed version. In practice, it does encounter a small percentage of decompression failures.

Smart Caching: Our crawler acts very much like a web cache. Once we grab your content, we hold onto it and keep a history of how it changes over time. We do this for a variety of reasons. One of them is so that we can use HTTP mechanisms designed to help reduce network usage when a client (that’s us) repeatedly fetches a web file that has not changed. In particular, our crawler often sends the HTTP If-Modified-Since header (see section 14.25 of rfc 2616) when making repeat requests. If your webserver is setup to recognize this header, it will respond with a 304 HTTP status code instead of a 200 if the content is unchanged. The advantage of this is that a 304 doesn’t include your page content, so it uses up less bandwidth than a full 200 response. Again, I’d like to emphasize that our crawler is conservative when it comes to ensuring it has the latest content; it won’t use an If-Modified-Since request if it needs to re-fetch your content for any reason.

Most webservers will automatically handle If-Modified-Since requests for static content out of the box. Proper cache control of dynamic content (such as PHP pages and cgi scripts) can be tricky and is an advanced topic. In most cases, servers will play it safe by ignoring If-Modified-Since requests for dynamic content. There are several sites on the web that let you test the cacheability of your web pages. For the purposes of our crawler, pay attention to what they say about the Last-Modified value in your response header.

Crawl-Delay: There’s one last trick you can use to help reduce the bandwidth requirements of your site. You can use a special robots.txt directive, crawl-delay, to reduce the speed at which our crawlers make requests to your site. This allows webmasters to manage their bandwidth without restricting content on their site from crawlers and is being used effectively by sites like Slashdot. A safe value for this would be a delay that would allow us to fetch every page on your site in about five days. So a five second delay (crawl-delay: 5.0) would be fine for a site with 2,000 pages, but not for a site with 100,000 or more.

We hope you find these tips for safely saving hosting bandwidth useful and we’d appreciate any feedback, questions or new ideas to further help improve how our crawler interacts with your web sites.

Dave Simpson
Yahoo! Search Engineering

Search Keys is a great little Firefox extension that makes navigating search results easier from the keyboard. I couldn’t stand to see this cool functionality not yet available for Yahoo! Search, so I built a quick Friday Hack (more on that tradition later) last week…

Here’s what you need to do to try it:

1. First, download the Search Keys extension for Firefox.
2. Shutdown Firefox.
3. Search your profiles extensions folder for
   search-keys.jar and replace it with this one.
4. Restart Firefox and do a Yahoo! Search.

You should now get little numbers
in boxes next to your Yahoo! Search results. Press the number to jump to
that result.

Once I know this is working well and we’ve gotten all the bugs out, I’ll
send a patch to the Search Keys maintainer. So let me know what you think.

Marc Abramowitz

Technical Yahoo

To celebrate the public launch of RSS on Yahoo! News Search I thought I’d write a quick blog entry and expose some of the more useful (and lesser known) features of the service.

We have over 7,000
sources and searching them is monkey easy; just go to search.yahoo.com/news (or news.yahoo.com) and type into the search box.’But if you want to get a bit more advanced and make very focused queries, you can do some really cool stuff.

Here are a few things to try.

Advanced Searches

By Location

Curious what the Canadian press thinks about the Iraq war? A search for iraq
location:canada queries only news sources in Canada.

By Source

Interested in the BBC’s coverage of the British Prime Minister? Just search for source:bbc blair.

By Category

Want to keep an eye on your home team? Try cat:sports houston if you live in Houston.

By Language

How about news in German? (Notice the “vl&=lang_de” parameter in that URL?)

By News Type

For all press releases involving IBM, use the newstype option: newstype:pr ibm.

More Advanced Options

You can compose your own advanced search by using the above search commands
or use our advanced news search page.

RSS Support

Two other cool features of Y! News Search are RSS support and the
ability to add any search to My Yahoo!:

• Use the ‘Add to My Yahoo!’ button on the search result page to add any
  news results you want to your My Yahoo! page.
• RSS: using the new xml button on the News Search result page, you can now
  turn any search into a fresh RSS feed. For example hurricane
  location:florida gets you your own ongoing feed of hurricane news
  from Florida. The only restriction is that the source parameter doesn’t
  work in RSS mode, and some providers may choose to opt out of RSS
  results.

Finally, it’s worth noting that the feeds are now full-blown RSS 2.0
rather than the old RSS 1.0 RDF format that we first used.

Enjoy the new Y! News Search, and let us know what we can do to make it
better.

Jacob Rosenberg

Technical Yahoo!