How does a search engine find websites?

1. Leo types a question: which animals fly at night? There are billions of web pages. Nobody could read them all in time. So how do answers appear in a blink?
2. Long before Leo asked, a program called a crawler visited page after page. It reads a page, then follows the links on it to find more pages.
3. From every page the crawler notes the words. For each word it keeps a list of the pages that use it. This giant word list is called the index.
4. Now the search is easy. The engine opens the list for night and the list for fly. Pages on both lists match the whole question: the owl page and the bat page.
5. Often thousands of pages match, so the engine sorts them. It checks how well each page fits the words and how many other pages link to it. The best go on top.
6. The results appear in a fraction of a second, because the hard work was done earlier. The top result is not always right, so read carefully and ask a grown-up if you are unsure.

Tap each part: the web on the left, the search engine on the right.

• Web pages. A web page is one document on the web. Each of these six pages is about an animal, and its words tell you what the page is about.
• Links. A link takes you from one page to another. Links join the pages together like a web, which is how the web got its name. The crawler travels along them.
• Crawler. The crawler is a program that visits pages all day and night. It copies what each page says, then follows the links to pages it has not seen.
• The index. The index has an entry for each word, with a list of every page that uses it. It works like the index at the back of a book, only far bigger.
• Search box. You type your words here. The engine splits your question into words and looks each one up in the index.
• Ranked results. The matching pages are sorted so the most useful ones come first. Pages that fit your words well, and that many other pages link to, move to the top.

Tap pages to send the crawler. Watch their words drop into the index. Then pick a search word and see which pages match.

Experiment: The crawler has not read a page yet. Can a search find that page?

1. Tap two or three pages for the crawler to read.
2. Pick a search word. Look for pages marked with a question mark: they match, but the index does not know them.
3. Send the crawler to those pages and watch the results change.

What we found: A search can only find pages the crawler has already read. Before the visit, the page matched your word but stayed hidden, because the search looks in the index, not on the web itself.

Turn on JavaScript to try it yourself.

The crawler's battery lasts for only 4 visits. Choose pages so that a search for night finds 3 pages and a search for bird finds 2.

When you search, the answer comes back before you can blink. That is only possible because the search engine did its homework long before you asked. It has three jobs: crawling, indexing and ranking.

Crawling means visiting pages. A program called a crawler reads a page, saves what it says, and then follows the links on that page to find more pages. Crawlers work all day and all night, because the web keeps changing.

Indexing means making a giant word list. For every word, the index keeps a list of the pages that use it. It is like the index at the back of a book, where you look up a word and find page numbers. When you search for two words, the engine looks at both lists and keeps the pages that are on both.

Ranking means putting the best pages first. The engine looks at clues: do your words appear in the title, how often do they appear, and how many other pages link to this one? The top result is the engine's best guess, not a promise that it is true. Check more than one page, and ask a grown-up when something looks odd.

Key words

Crawler

A program that visits web pages and follows their links to find more.

Index

A giant list that says, for each word, which pages use it.

Query

The words you type into the search box.

Link

A connection from one web page to another that you can follow with a tap or click.

Ranking

Sorting the matching pages so the most useful come first.

Posting list

The list of pages stored in the index for one word.

Check your thinking

1. Why can a search engine answer so quickly?

   Show the answer

   It already made a word list of the pages before you asked. The crawling and the word list are done ahead of time. Searching is just looking up words in that list.

2. A brand-new page was put on the web one minute ago. Why might a search not find it yet?

   Show the answer

   The crawler has not visited it, so it is not in the index. A search looks in the index. Until the crawler reads the page and adds its words, the engine does not know the page exists.

3. The list for swim has the duck, fish and frog pages. The list for bird has the owl and duck pages. Which pages match a search for swim bird?

   Show the answer

   Only the duck page. The search keeps the pages that are on both lists. Only the duck page has both words.

4. Two pages match a query equally well, but far more pages link to the first one. Under a rule that counts links, what happens?

   Show the answer

   The first page ranks higher. Link analysis treats each incoming link as a vote of confidence, so the page with more links gets the higher score.

5. The top result of a search says something surprising. What is the smart thing to do?

   Show the answer

   Check another page or ask a grown-up. Being first means the engine guessed the page is useful, not that it is true. Checking a second page is a good habit.