Chapter 1 - Inside the guts of a search engine

For simplicity’s sake, let’s say there are three pieces of software that together

make up a search engine - the Spider software, the Index software and the

Query software.

If you understand what these three do, then you have the foundation for

getting your website to the top of the search engines.

Here’s what the three types of software do:

The Spider software ‘crawls the web looking for new pages to collect and

add to the search engine indices’.

This is a metaphor. In reality, the spider doesn’t do any ‘crawling’ and

doesn’t ‘visit’ any web pages. It requests pages from a website in the same

way as Microsoft Explorer, or Firefox or whatever browser you use requests

pages to display on your screen.

The difference is that the spider doesn’t collect images or fancy designs - it

is only interested in text and links AND the URL, (for example,

http://www.Unique-Resource-Locator.html) from which they come: it

doesn’t display anything and it gets as much information as it can in the

shortest time possible.

A spider loves links because they lead it to other web pages that have the

things that it loves, guess what? more text, links and URLs!

The Index software catches everything the Spider can throw at it (yes,

that’s another metaphor). The index makes sense of the mass of text, links

and URLs using what is called an algorithm - a complex mathematical

formula that indexes the words, the pairs of words and so on.

Essentially, an algorithm analyses the pages and links for word combinations

and assigns scores that allow the search engine to judge how important the

page (and URL) might be to the person that is searching. And of course it

stores all of this information and makes it available to people who are

searching.

The Query software is what you see when you use a search engine - it is

the front end that everybody thinks of as a search engine. It may look simple

but it presents the results of all the remarkable search engine software that

works away invisibly on our behalf.

The main feature of the query software is the box into which people type

their search terms.

Type in your words, hit search and the search engine will try to match your

words with the best web pages in can find through searching the web.

But this too is a metaphor and perhaps the most important one.

The query software doesn’t search the web - it checks the records that have

been created by its own index software. And those records have been made

possible by the raw material the spider software collects.

What you need to understand about search engines

What you need to understand is that the search engine has done all the hard

work of collecting and analyzing web pages, BUT it only makes that

information available when someone does a search by entering words in

the search box and hitting return.

The words people use when they search therefore determine the results

the search engine presents. We call them keywords - that might sound fancy

but keywords are only ‘the words people use when they search’.

And keywords are what Wordtracker provides - many millions of them. Use

keywords in your website copy and you will prosper: ignore them and your

online business will surely perish.