How Search Engines Work : Learn About Search Engines & Their working

I’ve decided to start this year by revisiting the chapter in the book about how search engines work. I’ve said many times over the years that, most books about SEO have a section called how search engines work. But rarely (if ever) do they describe the interdisciplinary approach to information retrieval (IR) covering mathematics, computer science, library science, information architecture, cognitive psychology, linguistics, and statistics – to name but a few.

Previously, I had written mainly about methods of manipulating rankings by keyword stuffing and other black hat type techniques of the time. But as I began to realize the importance of linkage data and even more so, link anchor text, I became more and more inquisitive as to what it was exactly that search engines used in their ranking technologies.

After talking to one of the pioneers in web search (Brian Pinkerton of WebCrawler) I was introduced to the work of foremost information retrieval scientist, Gerard Salton. This was a major breakthrough for me.

Salton’s work was cited in just about every IR research paper I read at the time. So I tracked down and bought a copy of his seminal work “Modern Information Retrieval” (written back in 1983, however Salton’s work in the field goes back to the early 1970s).

As a marketer, not a scientist, this was no easy read. Yet, as I began to grasp the basic concepts and drivers behind information retrieval (and the way it is applied to the web) the more I was able to understand the major challenges involved. And that led me to change, not just the amateurish and spammy techniques I’d used previously, but to thinking about SEO in an entirely different way.

And to this day, I still firmly believe that a basic understanding of the science of information retrieval on the web goes a long way toward helping search marketers dispel myths and do their jobs more professionally and proficiently.

Of course, 10 years later my personal library has grown to include a very large section of information retrieval and data mining texts as more and more become available. This is also largely due to the fact that the subject matter is so fascinating it’s hard not to become engrossed.

As I revisited the chapter I’d written on how search engines work from a decade ago, I expected it to be a bit stale, but it wasn’t at all. Although, I dare say, to an IR scientist, if not stale it probably seems about as elementary as it gets. I wrote the chapter placing great emphasis on trying to make it non-mathematical. By that I mean highlighting concepts and background theory rather than matrices and formulae. That said, it’s extremely hard to cover the subject without references to linear algebra such as eigenvalue decomposition and other mind-numbing math.

Anyway, if you’re genuinely interested in how search engines work (but really, not the anecdotal stuff generally bandied around) then it’s as good a place as any to start. I mention in the introduction that it is totally unchanged from the very quirky, very British flavor it had when it was first published. A few pages were eliminated purely because they were totally irrelevant a decade later. There are a few little gems in it which I’d forgotten about.

? !

No the subhead above isn’t a typo or spelling mistake. It’s actually a conversation.

When the French author Victor Hugo had Les Miserables published, he was not living in Paris at that time. He was waiting to hear news from his publisher about the kind of reception his new book was having. When he could wait for news no longer, he sent a letter to his publisher which contained only the character: ?

On receiving this, his publisher knew exactly what it meant and he returned a note to him containing only the character: ! This let Victor Hugo know that his book was a huge success. It is said that this is the shortest correspondence in history.

What’s that got to do with anything? It was actually a good analogy I used relating to the length of the average query at search engines at the time and how difficult it is to deal with short queries.

I’m seriously thinking about trying to find the time to update the entire book this year and make it available free to Search Engine Watch and ClickZ subscribers. More recently, I’ve been reading about a feature-centric view of information retrieval, learning how to rank them, and about natural language processing that tends to use text annotation tools to label data (very hot research topics). I’ll be writing a couple of follow-up columns covering these subjects combined with fascinating insights into the strength of end-user data and, of course, weaving that into the update of the book once I get time to make a start.

But right now, feel free to download the PDF of “How Search Engines Work”. If nothing else, I hope it acts as a very basic introduction.