How do search engines work? Crawling, indexing, ranking

Search engines work in three stages, done long before you type. Programs called crawlers visit pages across the web and follow their links. An index then records what each page is about, like a vast card catalog. When you search, the engine doesn't search the web at all: it looks up its index and ranks the matches, using hundreds of signals, in a fraction of a second.

In short

• Crawling finds pages, indexing records what's on them, and ranking decides the order you see them in.
• A search runs against the engine's own copy of the web, built in advance, which is why it is so fast.
• AI-written answers now sit above the links for many searches, changing how often people click through.

How is a search engine like a library?

Picture a library with no front desk and hundreds of billions of books that change every day. Search engines solve the problem a librarian would. First, someone has to walk the shelves and collect every book, including the new ones: that is crawling. Next, someone has to write a catalog card for each, saying what it covers: that is indexing. Finally, when a visitor asks a question, a librarian has to pick the ten books most worth handing over, out of thousands that mention the subject: that is ranking.

The analogy also fixes a common misconception. When you search, you are searching the engine's catalog, not the live web. A page the crawler hasn't visited yet can't appear, and a page that changed this morning may still show yesterday's version. The web being searched is the World Wide Web, the pages and links that run on top of the network; the internet and the web are not the same thing.

How do crawlers find pages?

A crawler is a program that downloads a page, reads it, and adds every link it finds to a list of pages to visit next. Google's is called Googlebot. Starting from pages it already knows, it follows links outward, which is how most new pages are found. Site owners can also hand over a sitemap, a file listing their pages.

Crawlers are polite by design. Google's Search Central documentation says its crawlers try not to visit a site so fast that they overload it. A site can also post a file called robots.txt telling crawlers which parts to skip, a convention from 1994 that was finally written up as an internet standard, RFC 9309, in 2022. Pages behind a login aren't crawled at all. Modern crawlers also run a page's JavaScript, as a browser would, because many sites build their content in the browser rather than sending it ready-made.

What is an index, and how big is it?

The index is the catalog. At its heart is a structure called an inverted index: instead of listing the words on each page, it lists, for each word, the pages it appears on. Here is a toy one for three short pages:

Word	Pages it appears on
router	Page 1, Page 3
wifi	Page 1, Page 2, Page 3
slow	Page 2, Page 3

A search for "slow wifi router" looks up three short lists and finds the page on all three: Page 3. No page has to be read at search time, which is how an engine answers in well under a second. Real indexes also record where on the page each word appears, in the title or a heading or the body, and how often.

The scale is hard to picture. Google says its index covers hundreds of billions of webpages and is well over 100,000,000 gigabytes in size. Not every page gets in: indexing groups near-identical copies and keeps one, the canonical version, and leaves out pages it judges low quality or that ask not to be indexed.

How do search engines decide what ranks first?

Ranking is where engines differ most, and where they say least. Google describes "hundreds of factors", which fall into a few families: what the words in the query mean, including synonyms and typos; how relevant a page's content is; how trustworthy and useful the page seems; whether it loads fast and works on a phone; and context such as the searcher's location, language and device. Someone searching "football scores" in London and in Dallas wants different sports. Settings and search history, often tied to cookies or a signed-in account, can personalize results further.

The idea that made Google, in a 1998 paper by Sergey Brin and Larry Page at Stanford, was to treat a link as a vote, and a vote from an important page as worth more. They called the score PageRank. A toy web of three pages shows how it works. Page A links to B and C; B links to C; C links only to A. Each page passes its score evenly along its links, plus a small share every page gets regardless, and the passing repeats until the numbers settle:

Page	Links pointing to it	Settled score
C	2 (from A and B)	0.40
A	1 (from C)	0.39
B	1 (from A, shared with C)	0.21

A has only one incoming link but nearly ties C, because its one vote comes from C, the most important page, which gives A its whole share. B also has one link, but A splits its vote between B and C, so B gets half as much. Link analysis is now one signal among hundreds, but the idea still matters: a link from a respected site counts for more than many from unknown ones.

Two things are separate from ranking. Ads are sold and labeled as ads, and Google says it doesn't accept payment to crawl or rank a site higher. And search engine optimization (SEO), the practice of making pages easier to find, runs from the helpful, such as clear titles and fast pages, to spam, which engines' rules penalize.

How is AI changing search results?

For many searches, the first thing on the page is now an answer, not a link. Google launched AI Overviews, short answers written by a generative AI model with links to their sources, for all US users on 14 May 2024, and other engines and chat assistants offer similar answers. The model still depends on the three stages: it writes from pages that were crawled and indexed. What changes is whether people click through to those pages. Artificial intelligence here sits on top of search rather than replacing it.

Still argued

Whether AI answers starve the websites they draw on is disputed. A Pew Research Center study of 900 US adults' browsing in March 2025 found that people clicked a regular result on 8% of visits to search pages with an AI summary, against 15% without one, and clicked a link inside the summary on just 1%. Google's head of Search, Liz Reid, wrote in August 2025 that total clicks from Google to websites were "relatively stable year-over-year" and that clicks had become higher quality, meaning fewer quick returns; publishers and critics noted that Google published no data to back this. The two measure different things, and neither settles it.

Search still rests on the same plumbing that carries every page: crawlers fetch over the internet just as your browser does.

Check yourself

• Without looking back: why does a search come back in under a second, when the web has hundreds of billions of pages?
• What is an inverted index, and how would it find a page for a three-word search?
• In the three-page example, why does page A score almost as high as page C with only one incoming link?

Sources

• In-depth guide to how Google Search works — Google Search Central
• Organizing information: how Search works — Google
• The anatomy of a large-scale hypertextual web search engine — Sergey Brin and Lawrence Page, 1998
• Generative AI in Search: let Google do the searching for you — Google, 2024
• Google users are less likely to click on links when an AI summary appears in the results — Pew Research Center, 2025
• AI in Search is driving more queries and higher quality clicks — Google, 2025