As promised, this is the second half of the Interview with Tim Converse.

JQ: There's been this sort of continuum evolving since the Alta Vista days, where we started with primarily static content and then the next generation was catalogue type shopping like Amazon. Now it's this whole micro-publishing independent thing. What do you consider the biggest challenge right now in terms of classifying content?

A: One of the big challenges for us is just understanding what's out there. It's almost like astronomy where you're just trying to catalogue all the different things out there and track how they're growing, to some extent. So it's important for us to know just how many sites and kinds of documents there are so that we can catch trends.

JQ: You talked about comprehensiveness. There's this perception that there's the web that most of us see and then this dark web: the stuff that the crawlers don't reach. How do we try to get that data into the index? Are there barriers that webmasters put up that they should avoid to help us better index the content?

A: At it's simplest, webmasters aren't aware of robots.txt and it's uses. Redirection can also be problematic if people create content by creating lots of domains or hosts so we encourage people to organize their sites in many documents before they get a new host.

And of course, there's also the issue of crawler traps which some people do intentionally but much more often, they've unintentionally created crawler traps....

JQ: ...and a crawler trap is...

A: A crawler trap is something where you crawl a page and it has a link, usually in the same site that's dynamically created and then you follow that link and it has another analogous link that's dynamically created and often, just because people make mistakes, you're attaching on another directory every time which doesn't exist and takes you back to an automatically generated error page which has the same link. So you can fall into traps where there are an infinite number of pages that don't have any content.

Another thing people can do to help us is, this is sort of geeky but, don't make page not found pages that return a status 200.

JQ: I was just about to ask that. 404 pages back in the day, were these ugly grey things with block text that all looked the same and now they're done up to look like regular pages to be more appealing to users.

A: We do actually have ways of detecting that but it's a lot easier for us if a web server just says, "this page doesn't exist" as opposed to creating a nice page for the user that to a crawler looks like any other page. In general, if the server tells us 404, then we discard it.

YQ: I worked for a company that used CIDs instead of cookies to follow users through the site and it turned out to be a disaster. We went from having pretty much every page indexed to hardly any. So what about CIDs and how they affect the crawlers?

A: If you have differences in the URL that don't actually make a difference in the site, that can be hard for us to untangle. We're getting better at it. One of the scenarios you're talking about there would just create a lot of duplicates for us. So it's nicer for us if we have one URL per actual content but we understand that you're not designing this just for us. And we obviously do a lot of duplicate detection--actually, we do duplicate detection in a couple of different ways. Finding out if documents are the same; finding out if sites are mirrors of each other.

JQ: This question came up today on a mailing list that I'm on. The concern for this particular company is that they want to move their site to a new domain but they don't want to become invisible for the next six months or year or however long it'll take for people to point to their new website. What can we tell people like that?

A: We can tell them that in the future, if you actually want to move your site, you want to use a 301 redirect which will do as much of the right thing as we can.

YQ: What actually happens there? I've heard of companies who have used 301 redirects and yet their old pages continued to show up in the search engines anyway. Why is that?

A: The underlying problem is that people out there haven't changed their links and search engines do pay attention to links.

I can't give you a date, but we're changing how we deal with redirects. The thing about redirects is that everyone thinks it's obvious how a search engine should treat them and the obvious answer is not really that helpful. Any policy you develop with redirects is going to make someone unhappy but what we're about to roll out we will pay better attention to 301 redirects and the exact problem you're talking about should be less.

[In the time since we met with Tim, the team has rolled out a fix for 301/302 redirects. Documents will be handled by the new redirect policy as they are re-crawled and re-indexed and webmasters will start to see many of the sites change in the next couple of weeks. The index should be fully propagated within a month. See Tim Mayer's Webmaster World presentation for details.]

YQ: You mentioned earlier that you'd just bought a piano. I read on your website that when you were eight, you ran away from home to escape piano lessons. Is that true and did you just hate piano back then?

A: Yeah, that's true but I never hated piano, just lessons. Now that I've bought the piano, I'm practicing again. Right now I'm learning a classical piece by Bach. I'm a slower learner now though--I should have stuck with it when I was eight. But when it comes to the kind of music I actually listen to, I like rock, hip hop and classical. I'm not too into jazz.

JQ: Which do you listen to when you're programming?

A: (laughs) I don't actually. I don't deal with headsets well.

JQ: In terms of freshness, there's a lot of talk about how quickly an RSS-watching engine will pick up new content as opposed to getting stuff into Yahoo! Search. The question they ask is why can't we just ping Yahoo! and get the crawler over here?

A: Well, that's not the only source of latency or possible delay. We build very large databases and it's kind of a large industrial process involving lots and lots of machines. There's some delay between the last document we heard about and the time we actually put something live. In some cases that delay matters more than any delay in finding out if something has changed. We also pay a lot of attention to whether something has changed. But I think you'll see us getting fresher and fresher.

YQ: When you're looking for things that are changing on the page, what are you specifically talking about? I'm sure it's not enough to just change a hidden date stamp in a footer.

A: Yes, It's more than that. Most of the web is just static even without there being date stamps. We do have a more nuanced notion of what it means to change so we can detect a trivial change from a significant one. We can tell a major change from a trivial one.

YQ: You mentioned that you were between the scientist and the programmer. What do you think you uniquely bring to what you're doing?

A: Well I'm hoping that one side thinks I'm the other side and that the other side thinks I'm the one side. (laughs) I'm hoping I've got them all fooled but I have this feeling that I don't. But no, I think that what I uniquely bring is that I can talk to both sides, I've been a programmer; I've trained to some extent in the direction that the scientists have trained and went to grad school for a long time in related topics so increasingly, though I never thought I would play this role and it's not what I envisioned, what I'm bringing to it is I can talk to a lot of different sides and I can prioritize the stuff and my grasp of the technology's not so bad either.

Yvette Irvin
Y! Profiler