A while back, Andrei Broder, a Yahoo! Research Fellow and Vice President of Emerging Search Technology, spent an afternoon telling us a bit about his decades-long history within the search industry and talking about his future projects. To wrap up our interview, we close with some of Andrei�s observations on several Yahoo! Search blog reader questions.

So, several people asked how you feel about what happened to AltaVista�

AltaVista had almost perfect bad timing; it started with huge technology advantage but an unsustainable business model at that time, and squandered its early lead in core search competency.

One of the questions from a reader was about your taxonomy paper. Can you talk a bit about that?

In there, I talk about the three stages of search, that I mentioned before. Web search started in early-mid nineties, really as a scale-out of the classic information retrieval model. At that time, people were still trying to find the best way to adapt classic information retrieval to the scale of the Web: Boolean models, Probabilistic models, etc. The second phase, in the late 90s, was about metadata. Hyperlinks, labels, clickthru data, all sorts of metadata of any kind. The structure of the web. But it was still very syntactic in nature, basically matching words against text. There is no understanding of meaning here. The third generation, still in progress, is about text semantics and analysis, where you starting to understand what the queries are about. That�s roughly where the paper stops . And now there are things like Yahoo! Shortcuts, or a lot of the information that is derived from the meaning of the query. Semantic, shortcuts, local search all seems to be taking off. So it seems that the paper was correct in predicting semantic search as the next phase at the time. Of course if I were to expand the paper now, I would write about the fourth phase: information supply.

Have you looked at blog search? Why does it �suck�?

Blog search is difficult. If you look at web search in general, the biggest help comes from metadata, anchor text, links, web graph analysis, etc. For blogs we have very little useful metadata. And even if you do have metadata for blogs, it is often wrong, or you can�t trust it, so you use it very little.

Furthermore, context is not always there. A lot of blogs are not self-contained, context surrounds them. Even a human doesn�t understand what�s going in a blog if dropped into the middle of it. I�m not sure how much progress we will see, (but then again, I�m not focusing on that!)

Finally, there were some questions about spam.

Any kind of information signal one might use in a search engine, spammers will try to pollute it. We need to be careful not only about link spam, fake sites spam, and so on but also about pollution in the query log, and other more subtle sources of information. On the other hand, spamming is an economics game, people think spammers are kids up to no good, but it is not. Spam is about economics, and we want to raise the difficulty of spamming to the point where it�s not economic to spam. As we go to a more personalized search experience, social aspects of search will play an increasing role. It remains to be seen to what extent this is spamable � It is hard to make robots that behave as humans, it might well be the case that social aspects of search are fairly spam resistant.

Thank you so much for your time, Andrei, and to all our readers. Please leave us a comment below and let us know what you thought!