Google's Internet Search Guru Breaks Cover

The man who oversees Google’s fabled search algorithms that determine the all-important page rank and search results on the world’s most powerful search engine – a business asset as closely guarded as Coca-Cola’s “secret formula” or Colonel Saunders’ “secret recipe of 21 different herbs and spices” – has emerged from the relative seclusion of Google’s Mountainview, California headquarters to give some unheralded yet noteworthy insights into how Google’s search function operates.

The secrecy that surrounds Google’s all-important search algorithms is understandable, for it is these algorithms more than any other of Google’s many assets that form the core of its business – and Google’s “Don’t Be Evil” ethos dictates that it guards its algorithms from the hackers and gamers who would quickly undermine the ability of Google to deliver quality search results to the hundreds of millions of users that rely on Google to access information on the world-wide web.

“The details of the ranking algorithms are in many ways Google’s crown jewels,” writes Udi Manber, Google’s VP Engineering – Search Quality and the man entrusted with safeguarding and refining Google’s search equation, in his recent post on Google’s official blog (http://googleblog.blogspot.com). “By some estimates,” he notes, “more than one thousand programmer/scientist years have gone directly into their development and the rate of innovation has not slowed down.”

In one of his first publicized writings for Google (he was for many years a computer science professor at the University of Arizona, before moving to Yahoo!, Amazon, and then Google), Manber comes clean on why so little is known about the algorithms that catapulted Google into its position of search engine dominance, propelling the verb “google” all the way into the Oxford English Dictionary. “For something that is used so often by so many people, surprisingly little is known about ranking at Google,” the Google search engineer admits, noting that the close-vestedness with which Google guards its algorithms is designed to forestall both competition and abuse.

“Competition is pretty straightforward,” he observes. “No company wants to share its secret recipes with its competitors. As for abuse, if we make our rankings formula too accessible, we make it easier for people to game the system. Security by obscurity is never the strongest measure, and we do not rely on it exclusively, but it does prevent a lot of abuse.”

Abuse prevention is, of course, at the heart of Google’s business model. Search engine optimization (SEO) “gamers” devise endless varieties methods to get around Google’s algorithms in order to have sites they wish to promote rank higher for specific keyword terms on Google’s search results page than they otherwise would. The literature is replete with examples of what Google describes as “black hat” SEO practices. Part of Google’s Search Quality team function is to ensure that such practices do not prevail, and the end users of Google search (you and I sitting at our keyboard typing in our search terms) gets the results that are most relevant to the information we are looking for.

If end users find that they are not getting the most relevant results using Google, they will quickly switch to a search engine that gives them higher quality results. After all, it was Google’s page rank system and its algorithms (which are being ever perfected by Udi Manber and his team) that boosted Google to prominence over such earlier search engine also-rans as Lycos and WebCrawler. Mr. Manber is essentially tasked with seeing that does not happen. In one of his first public interviews, a recent interview in, of all places, Popular Mechanics magazine, he revealed that his team at Google made over 450 changes to the search engine giant’s algorithms in 2007 alone. And the necessity for and pace of such changes, only parts of which could be classified as “tweaking”, is only likely to increase.

The heretofore reclusive Googler likens the evolution and development of internet search to “science fiction” – what is seemingly fiction (the stuff of dreams) now will be our reality, just as routine manned space flight and probes to Mars and the distant outreaches of the solar system were a generation or three ago. “When the first search engine appeared in ’94, compared with when I came out of academia in ’99, compared with what it was in 2003, compared with the way it is today – every five years have been just incredible advances,” Google’s algorithm chief observes.

Sitting down with reporter Stephen Shankland of C/NetNews. Com, Mr. Manber was candid and excited about how Internet search has grown and will continue to grow. “In the early 1990’s, everybody was talking about the information revolution,” he says, “It was very clear that to have an information revolution, its not enough to store the information and move it around, you have to find it.” Now, he observes, it is expected that the user will be able to find the information he or she is looking for almost instantaneously, by merely typing in two or three words. Internet search was, in those early days, very hit and miss. ”Nowadays,” he observes, “if you don’t find exactly what you want in the first or second result something is wrong.”

This degree of consumer expectancy is a testament to how proficient Google, Yahoo! and the other search engines have become at what they do best – getting relevant information to people that are actively seeking that information out. And while this will always likely remain the core of the search engine model, refinements are in the pipe. “There are a lot of things you don’t search for now, because you don’t expect Google will know or that the search engine will find out,” the Google Veep notes. But, he says, “We are finding that user expectations grow. The kind of searches people do now are more complicated than the kinds they were doing five years ago. People expect a lot more from us.”

While Coca-Cola once famously muffed up by pulling the core line of its product off the shelves in the “New Coke” versus “Coca-Cola Classic” marketing fiasco, don’t expect Google to do the same by taking its eye off the core component of its business model: providing the most relevant search results to users of its services. But just as Coca-Cola came out with Diet Coke, Vanilla Coke and Coke Zero etc. to appease increasingly sophisticate palettes, and even Colonel Saunders is experimenting with vegan nuggets, expect that Google will keep rolling out the evermore sophisticated search products and search options it is researching and testing in Google Labs. Meanwhile, know that Mr. Manber et al. are working hard to protect the integrity of the Google algorithms, allowing ever better, increasingly refined organic search results, while trying to keep one step ahead of the posse of “black hat” SEO gamers who, in an ironic role reversal, are chasing the search engine giant that envisions itself wearing the “white hat”.