Live Testing at Yahoo! Search

My job reminds me of an online BBS game called Planet Busters I used to play about 15 years ago. In that game, every day I would look at the landscape of the universe, the condition of each of my planets, and the results of the previous turn, and give commands for the next turn: send 5 ships to colonize planet X, send 1 ship to scout out planet Y, send 100 ships to conquer planet Z. After committing my commands, I’d have to wait until the next day to see the results. Every afternoon logging onto the BBS was like a treat. I excitedly fired up the game — “Did my strategy work? Shoot, all 100 ships were destroyed?? My scout ship arrived and took over an undefended planet Y! Yes!”

At Yahoo! Search, I coordinate live testing for the Web search user interface (UI). Similar to Planet Busters, I plan out moves in the form of tests to try out on Web search. Even more exciting than the online game I used to play, however, is that if a test goes well, it results in a change to the Web search UI that helps people find what they are looking for even more accurately or more quickly than before.

Making changes on Yahoo! Search is analogous to changing the tires on a car going 100 miles per hour. We need to be able to make positive changes to the UI without stopping the service and without users ever missing a beat. Live testing allows us to do so. In a live test, we give a slightly modified UI to a small number of our users for a few days. We collect metrics on how these users interact with the test UI and then make inferences on whether the changes are good or not-so-good for users. Good changes are released for everyone. The not-so-good ones are brought back to the drawing board for refinement or modification.

Who knew that playing strategy games in my childhood would be training for my eventual job at Yahoo! Search?

Fred Delse
Product Manager, Yahoo! Search