How Search Engines Work: Crawling, Indexing and Ranking, Explained by Google's Own Rules

The life of one page, from publish to first click

Google describes its system in three stages: crawling, indexing and serving results, and says plainly that not every page makes it through all three. Follow one hypothetical page through the machine. Say a roofer in Boerne publishes a page about metal roof repair on a Tuesday morning.

1. 1

   Publish and link it

   The page goes live and gets a link from the roofing services page and an entry in the XML sitemap. Without at least one of those, Google has no route to it.

2. 2

   Discovery

   Googlebot recrawls the services page, finds a link it has never seen, and adds the new address to its list of URLs to visit. Discovery is not a visit. It is a note on a to-do list.

3. 3

   Crawl

   Googlebot requests the URL, mostly as a smartphone crawler, and the server answers with a 200 status and the HTML. A 500 error here tells Google to slow down and come back later.

4. 4

   Render

   Google runs the page in a headless Chromium browser so text and links added by JavaScript show up. This can happen seconds after the crawl or later, when resources allow.

5. 5

   Index

   Google reads the text, title, headings, image alt text and links, checks whether the page duplicates another URL, picks a canonical version and decides whether to store it.

6. 6

   Serve

   A homeowner nearby searches for metal roof repair. Google pulls matching pages from the index, ranks them for that person's location, language and device, and the roofer's page either appears or does not.

7. 7

   Check

   The roofer pastes the URL into URL Inspection in Search Console and sees whether Google has crawled it, which canonical it chose, and whether it is indexed.

Every step can fail, and every failure looks the same from outside: the page does not show up. Knowing which stage broke is most of the diagnosis.

Discovery: there is no registry of the web

Google says plainly that no central registry of web pages exists. It finds pages in three ways: it already knows them from earlier crawls, it extracts a link from a known page to a new one, or a site owner submits a list of URLs in a sitemap. The vast majority of pages in Google's results, by its own account, were never manually submitted. They were found by following links.

That makes internal links the cheapest discovery tool a business owns. A service page linked from the home page gets found on the next crawl of the home page. A page with no inbound links is an orphan that depends on the sitemap. Our internal linking work exists because so many sites bury good pages four clicks deep.

Google's sitemap documentation says a small site, about 500 pages or fewer, that links its important pages from the home page may not need a sitemap at all. Large sites, new sites with few outside links, and sites heavy with video, images or news benefit most. A sitemap helps discovery, but Google says it does not guarantee that anything in it will be crawled or indexed. Our guide to XML sitemaps covers format and submission, and sitemap errors covers the usual breakages.

Crawling and rendering: what Googlebot actually downloads

Googlebot decides by algorithm which sites to crawl, how often, and how many pages to fetch from each. Google says it tries not to overload a server, and on most sites it should not request pages more than once every few seconds on average. Server responses steer it: errors and slow answers make it back off, which is one reason a struggling host can quietly slow the indexing of new pages.

Most crawl requests come from Googlebot Smartphone, so Google mainly judges the phone version of a page. Googlebot reads the first 2MB of an HTML file, a ceiling few ordinary pages come near. Pages it cannot reach at all are the common failure: a robots.txt rule left over from a staging site, a login wall, a server that times out. The fix guides for robots.txt blocking Google and Google crawl errors walk through each.

Then comes rendering. Google runs the page in a recent version of Chrome, executes its JavaScript, and indexes the result. Its JavaScript guide says a page may wait in the render queue a few seconds or longer, and still recommends server-side rendering or prerendering because not every bot runs JavaScript. That warning matters more each year: Vercel's December 2024 analysis found crawlers from OpenAI, Anthropic and Perplexity did not execute JavaScript. Our guides to JavaScript SEO and JavaScript and AI crawlers go further.

Indexing is a filing decision, not a right

Indexing is where Google reads what it fetched: the text, key tags such as the title element and alt attributes, images and video. Then it asks whether this page is a duplicate. Google groups pages with similar content into clusters, picks the most representative one as the canonical, and treats the rest as alternates that may show in other contexts. It also records signals about the canonical, such as its language, the country it serves and its usability.

The canonical choice is why a store with the same product at five URLs, or a site reachable with and without www, can watch Google index the wrong version. You can state a preference with a canonical tag, redirects and sitemaps, and Google usually listens. Our guides to canonical tags and redirects explain how to send one clear signal instead of five mixed ones.

Google is direct about the rest: indexing is not guaranteed, and not every page it processes will be indexed. The reasons it names are low-quality content, robots meta rules that forbid indexing, and site designs that make indexing hard. Even an indexed page may never appear if it is not relevant or good enough. Meeting Google Search Essentials only makes a page eligible.

Discovery is a note on a to-do list. Indexing is a judgment. Ranking is a contest held fresh for every search.

Crawled, not indexed: reading Google's verdict on your pages

Search Console's Page indexing report tells you where each URL stalled, in Google's own terse labels. Here is what the common ones mean. Our guide to Google Search Console covers setup and the other reports.

Google says most sites have at least a few unindexed pages and that you should not expect every URL to be indexed. What matters is that the important ones are. Crawl budget, the subject of endless forum threads, is a concern Google aims at sites with roughly a million pages changing weekly or ten thousand changing daily. If your pages get crawled the day you publish them, Google says you can skip that guide. When a whole site is missing, start with website not indexed by Google.

Ranking: hundreds of signals, and a few systems with names

When someone searches, Google looks through the index for matching pages and returns the ones it judges most relevant and highest quality. It says relevance depends on hundreds of factors, including the searcher's location, language and device. A search for a plumber in San Antonio and the same words typed in Boston return different businesses, by design.

Google's guide to its ranking systems names several of the machines involved. These are the ones a business owner hears about most:

Nobody outside Google knows the weights, and anyone selling a precise formula is guessing. Our guide to Google ranking factors sorts confirmed from myth. For trust, see E-E-A-T, which Google says is not itself a ranking factor.

How Bing does it differently

Bing runs the same three stages with its own crawler, Bingbot, which Microsoft says crawls billions of URLs a day and prioritizes known pages that are not yet indexed or have likely changed. Bing matters more than its search share suggests, because Microsoft says Copilot is powered by Bing's index, so a page missing from Bing is missing from those answers too.

IndexNow's own site lists Bing, Naver, Seznam.cz, Yandex and Yep as participants, not Google. Verify the site in Bing Webmaster Tools, submit the sitemap you gave Google, and see our page on Microsoft Copilot for how that index feeds AI answers.

Where AI Overviews and AI Mode sit in the pipeline

Google's AI answers are not a separate web. Its AI optimization guide says they use retrieval-augmented generation, or grounding, which relies on core ranking systems to pull relevant pages from the same Search index. AI Overviews and AI Mode may also use query fan-out, running several related searches across subtopics to build one response. To appear as a supporting link, Google says a page must be indexed and eligible to show with a snippet. There are no additional requirements or special optimizations.

So the pipeline above is also the pipeline for AI Overviews. A page that is not crawled, not indexed or blocked from snippets cannot be cited there. Google said in 2026 that optimizing for its generative AI features is still SEO. Clicks from those features count in Search Console's normal Web performance report, and since August 31, 2026 every site also has a Generative AI performance report showing impressions. Our page on Google AI Overviews covers the feature itself.

Other AI engines run their own crawlers, such as OpenAI's OAI-SearchBot for ChatGPT search. How they turn retrieved pages into a short list of named businesses is covered in how AI chooses which businesses to recommend, and the free AI visibility check shows what they say about you today.

What an owner should do with all this

The pipeline turns vague worry into a short checklist, one line per stage.

• Discovery: every page you care about is linked from at least one other page and listed in the sitemap.
• Crawling: robots.txt blocks nothing you want found, and the server answers quickly with a 200.
• Rendering: the main text and links are in the HTML, not added only after scripts run.
• Indexing: one URL per page, a canonical that agrees with it, no stray noindex, and content worth filing.
• Ranking: the page answers the search better than the pages above it, for the people and places you serve.
• Bing: the site is verified in Bing Webmaster Tools too, because Copilot depends on it.

If you would rather have someone check it, the free website audit tests speed, SEO, mobile, security and content and emails a report with fixes. Our technical SEO guide is the next read for doing it yourself, and the SEO guide puts this page in order with everything else. When the audit turns up more than an afternoon of work, our SEO service handles crawl, index and ranking problems as part of a monthly program.