Key takeaways
- Search runs in three stages: crawling (finding pages), indexing (understanding and storing them), and serving (picking and ordering results). Google documents these three stages directly, and a page can fail at any one of them.
- Being crawled is not the same as being indexed, and being indexed is not the same as ranking. Each stage is a separate gate.
- Crawlers find pages mostly by following links and reading sitemaps, then render the page like a browser, running its JavaScript.
- Ranking is decided at query time by hundreds of signals, weighted differently depending on the search, the person, and the intent.
- AI answers changed the last step, not the first three. Google still crawls and indexes the web, then an AI layer selects and summarises sources on top of it.
Most explanations of search engines are either hand-wavy (“Google reads your site and ranks it”) or drowning in jargon. Neither gives you a working mental model. This guide builds that model in plain language, then shows where AI answers fit into it. The mechanics below are described the way the search engines themselves document them, with sources you can check.
The single most useful idea to hold onto: a search engine does its work in distinct stages, and your page has to clear each one in order. Understanding those stages is what lets you diagnose why a page is not showing up, instead of guessing.
The three stages, in one view
Google, the most documented example, describes its process in three stages: crawling, indexing, and serving search results. Other engines differ in detail, but the shape is the same. Here is the whole pipeline before we go deeper on each part.
| Stage | What happens | What can go wrong |
|---|---|---|
| Crawling | Automated programs (crawlers or bots) discover URLs and download the page’s content. | The page is not linked anywhere, is blocked in robots.txt, or the server errors out. |
| Indexing | The engine renders and analyses the page, works out what it is about, and stores it in a database. | Content is thin or duplicated, or the page is marked noindex, so it never gets stored. |
| Serving (ranking) | At query time, the engine searches its index and orders the most relevant, highest-quality results. | The page is indexed but judged less relevant or lower quality than competing pages. |
Google is explicit that not all pages make it through each stage. That one sentence explains most “why isn’t my page showing up” problems. The fix depends entirely on which gate the page is stuck at, which is why the rest of this article treats them separately.
Crawling: how bots find your pages
There is no master list of every page on the web, so a search engine has to keep finding new and updated pages itself. Google calls this “URL discovery,” and it happens in a few ways: the engine revisits pages it already knows, it extracts links from known pages to reach new ones, and it reads sitemaps that site owners submit. The practical takeaway is that links are the road network of the web. A page that nothing links to, and that is not in a sitemap, is very hard to discover.
Once a URL is discovered, the crawler (Google’s is called Googlebot) may fetch the page. It uses an algorithmic process to decide which sites to crawl, how often, and how many pages to pull, and it deliberately throttles itself to avoid overloading a site. Google notes that for most sites Googlebot should not access the site more than once every few seconds on average, and that server errors like HTTP 500 are read as a signal to slow down.
Two technical details are worth knowing because they quietly shape what gets seen. First, crawling has size limits: Google says Googlebot crawls the first 2MB of a supported file type and the first 64MB of a PDF, then stops and only passes along what it already downloaded. Second, for most sites Google primarily indexes the mobile version of your content, so what the mobile crawler sees is what counts.
Crucially, modern crawling is not just reading raw HTML. During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome, the same way your browser does. This matters because many sites build their content with JavaScript. If that content only appears after scripts run, and those scripts are blocked or broken, the crawler can end up seeing an empty shell.
Pro tip. The lesson is simple: never block the CSS and JavaScript files your page needs to render.
Indexing: how the engine understands and stores a page
After a page is crawled, the engine tries to work out what it is about. Google calls this indexing, and it involves processing the textual content and key tags and attributes, such as the title element and image alt attributes, plus images and video. The output is an entry in the index, which is best pictured as the index at the back of a book: an enormous map of which words and topics appear on which pages. That index is vast. Google’s public explainer says its Search index contains hundreds of billions of webpages and is well over 100,000,000 gigabytes in size.
One part of indexing trips up a lot of site owners: duplication and canonicalisation. When Google finds pages with very similar content, it groups them into a cluster and picks the single most representative page, the canonical, to show in results. The others are treated as alternates. This is why a product available at five near-identical URLs does not get five entries. The engine collapses them and chooses one. If you do not tell it which URL you prefer, it will decide for you, and not always the way you would want.
The blunt truth here is that indexing is a filter, not a formality. Google states plainly that indexing is not guaranteed, and not every page it processes will be indexed. Low-quality or thin content, a stray noindex rule, or a design that makes the page hard to parse can all keep a page out of the index. If a page is not indexed, no amount of ranking work will help, because it is not even in the running.
Ranking: how the engine picks and orders results
Ranking, which Google files under “serving,” happens the instant someone searches. The engine searches its index for pages matching the query and returns what it judges to be the highest quality and most relevant. Google says relevance is determined by hundreds of factors, including things like the searcher’s location, language, and device. Its own example: a search for “bicycle repair shops” shows different results to someone in Paris than to someone in Hong Kong.
Two ideas do more than the rest to make ranking make sense.
First, ranking is contextual, not a fixed scoreboard. There is no single, permanent list of “the best pages for keyword X.” The result set is assembled per query, weighted for who is asking and what they seem to want. The same page can rank differently for two people searching the same words. This is also why the search results page itself changes shape by query: Google notes that a search for “bicycle repair shops” tends to trigger local results, while “modern bicycle” is more likely to surface image results. The engine is reading intent and adjusting the whole page, not just the ten blue links.
Second, relevance and quality are separate things. A page can be tightly relevant to the query yet lose to a more trustworthy, more useful competitor, and a very authoritative page can be beaten by one that matches the intent more precisely. Good ranking is the overlap of the two: closely matching what the searcher wants, from a source the engine has reason to trust. If you want the deeper mechanics of how relevance and authority get evaluated, that is a subject for the on-page and link-building guides in this cluster. For now, the mental model is enough: matched intent plus earned trust.
How AI answers changed the last step
The most common misconception right now is that AI answers replaced how search works. They did not. Crawling and indexing are unchanged. What changed is the final step, serving, where an AI layer increasingly reads the top sources for a query and writes a synthesised answer on top of the classic results. This is a large-scale shift, not a niche feature. Google reported that its AI Overviews had over 2 billion monthly users across more than 200 countries and territories and 40 languages as of its Q2 2025 earnings call.
Mechanically, the newer AI search experiences do something traditional ranking never did: they break one question into many. Google describes its AI Mode using a “query fan-out” technique, issuing many related searches at once and pulling from across the web to assemble a single answer. So instead of ranking one list for one query, the system runs a fan of sub-queries and stitches the results together.
What does this mean for a page? Two things. First, the underlying fundamentals still decide whether you are even eligible: if you are not crawled and indexed, you cannot be cited in an AI answer, because these systems draw on the same index. Second, the game at the top of the page shifts from “rank number one” to “be one of the sources the answer is built from.” You can be selected as a cited source without holding the top classic position, and you can hold a strong classic position and still not be quoted. This is why structuring content to be quotable, answer-first and self-contained, is becoming its own discipline. For a fuller treatment of how these systems differ from classic search, see our guide on what AI search is and how it differs from Google.
It is worth being precise rather than alarmist about the impact. Google has said AI Overviews are driving over 10% more queries globally for the types of queries that show them, and that its end-to-end AI Mode had passed 100 million monthly active users in the U.S. and India. The point is not that classic search is gone. It is that a summarising layer now sits in front of it for many queries, and that layer chooses its sources from the index the first three stages built.
Putting the model to work
The reason this staged model is worth internalising is diagnosis. When a page is not getting traffic, the useful question is never “how do I rank higher.” It is “which stage is this page stuck at.” Is it discoverable and crawlable? Is it actually indexed? Is it indexed but out-ranked? Each answer points to a completely different fix, and tools like Google Search Console report on each stage separately so you can tell which one is failing. Skip the diagnosis and you end up rewriting title tags on a page that was never indexed in the first place.
Frequently asked questions
Is being indexed the same as ranking?
No. Indexing means the page is stored in the search engine’s database and is eligible to appear. Ranking is a separate, later step where the engine decides whether and how high to show it for a given query. A page can be fully indexed and still rank on page five, or not appear for a query at all if it is judged less relevant than others.
How does a search engine find a brand new page?
Mostly by following links. Google discovers URLs by extracting links from pages it already knows and by reading sitemaps that owners submit. A new page that is linked from an existing, already-crawled page, or listed in a sitemap, will typically be found. A page with no inbound links and no sitemap entry is very hard to discover.
Does Google read JavaScript content?
Yes, within limits. Google renders pages and runs JavaScript using a recent version of Chrome during crawling. The risk is that if the files needed to render your content are blocked, broken, or too slow, the engine may not see the content those scripts generate. Making sure your CSS and JavaScript are crawlable is the safeguard.
Did AI Overviews change how pages get indexed?
No. Crawling and indexing work as before. AI answers operate at the serving stage, selecting and summarising sources from the existing index. Being crawled and indexed remains the prerequisite: if a page is not in the index, it cannot be surfaced or cited in an AI-generated answer.
Written by Matthis Duarte, a senior SEO and organic growth expert with 10+ years of experience driving organic growth for international brands across highly competitive verticals. He is the founder of Knownful, an independent publication on SEO and organic growth featuring in-depth guides, best practices, playbooks and original analyses, including a free monthly study of which brands AI actually recommends across 10 industries.
