Crawl budget: what it is, and when it actually matters

Few SEO topics generate as much anxiety, or as much wasted effort, as crawl budget. Owners of 300-page websites spend weekends worrying about it, while the businesses that genuinely have a crawl problem often do not realize it is happening. The honest answer is that crawl budget is real, it is well documented by Google, and for the large majority of sites it is something you can safely put out of your mind.

This guide sticks to what Google itself has published. We will define crawl budget the way Google defines it, explain why most sites can ignore it, look at what actually drains it in 2026, and cover how to protect it when you run a site large enough for it to matter.

What crawl budget actually is

Before a page can rank, or be quoted in an AI answer, it has to be crawled. Crawling is the first step in the pipeline of how search works, and it is worth understanding before you optimize for it. If you want the full mental model, we cover it in our guide to how search engines work. Crawl budget is simply the limit on how much of that first step Google will spend on your site.

In its crawl budget documentation, Google explains that the web is a nearly infinite space that exceeds its ability to crawl every URL, so there are limits on the time and resources it can devote to any single site. It defines a site’s crawl budget as the set of URLs Google can and wants to crawl, built from two components: crawl capacity limit and crawl demand.

Crawl capacity limit

Google does not want to overwhelm your servers, so its crawlers calculate a crawl capacity limit, also called hostload. This caps how much time your server spends holding connections open for Google, factoring in both the number of parallel connections and how long each one lasts. Per Google’s documentation, every site starts with the same conservative default, and the limit moves up or down based on crawl health: if your site responds quickly and consistently the limit rises, and if it slows down or returns server errors (5xx status codes) or rate-limiting signals (HTTP 429), Google crawls less.

Crawl demand

Capacity is what Google is willing to take from your server. Demand is how much it actually wants. Google says Googlebot’s demand varies with a site’s size, update frequency, page quality and relevance compared to other sites, and it names three factors you can influence:

• Perceived inventory. Without guidance, Google tries to crawl all the URLs it knows about. Google calls this the factor you can control the most: if many of those URLs are duplicates or unwanted, it wastes crawling time on your site.
• Popularity. URLs that are more popular on the internet tend to get crawled more often to stay fresh.
• Staleness. Google’s systems try to recrawl documents often enough to catch changes.

One definition detail matters here. Google’s crawling infrastructure treats a “site” as a single hostname, so www.example.com and shop.example.com are separate sites with separate crawl budgets. And a point of reassurance worth stating plainly: in its myths and facts about crawling resource, Google confirms that crawling is not a ranking factor. Crawling is necessary for a page to appear in results, but improving your crawl rate will not by itself lift your positions.

Why most sites can ignore it

Google is unusually direct about who should not worry about this. Its guidance opens by saying that if your site does not have a large number of pages that change rapidly, or if your pages seem to be crawled the same day they are published, you do not need to read the crawl budget guide at all. Keeping your sitemap current and checking the Page Indexing report is enough.

Google positions crawl budget management as an advanced topic for a specific set of sites, and offers these figures as a rough way to classify yourself (its words: these are estimates, not exact thresholds):

It also helps to kill a persistent myth. In its myths resource, Google addresses the belief that “small sites aren’t crawled as often as big ones” and marks it false: if a site has important content that changes often, Google crawls it often regardless of size. Smallness is not a crawling penalty.

So what do you do when a handful of pages on a small site are not getting indexed? In almost every case I have audited, the cause is not crawl budget. Google is crawling the pages fine and choosing not to index them, which is a quality and value judgment, not a resource limit.

What actually wastes crawl budget

When crawl budget is a genuine problem, the cause is rarely Google being stingy. It is usually a site handing Google millions of low-value URLs to chew through. The single biggest culprit is faceted navigation.

Faceted navigation is the filter-and-sort system on category pages: products, articles or events that visitors reorder by color, size, price and so on. Google’s documentation on faceted navigation URLs is blunt about the risk. Because each filter combination typically produces a new parameter URL, the common implementation can generate what Google calls “infinite URL spaces.” A single page can explode into a huge number of crawlable variants such as:

https://example.com/items?products=fish&color=green&size=tiny

Google describes two harms from this. First, overcrawling: crawlers cannot tell whether these novel URLs are useful without fetching them, so they access a very large number of them before determining they are useless. Second, slower discovery: time spent on useless URLs is time not spent finding your genuinely new, useful pages. On a large ecommerce or listings site, that is exactly how real content gets crawled late or not at all.

A few other common drains, and some things people wrongly worry about, are worth separating out:

That last row trips people up constantly. Because Google has to request a URL before it can read a noindex tag, slapping noindex on thousands of parameter URLs does not stop the crawling, and Google explicitly warns against using it to reallocate crawl budget. Over the long run, removing URLs from the index can indirectly free up crawling for other pages, but it is not the fast lever people expect.

How to protect it on big sites

If you genuinely run a site large enough to care, the goal is not to earn a bigger crawl budget. It is to stop wasting the one you have, so Google spends its time on the URLs that matter. Google’s own best practices come down to managing your URL inventory and keeping your server efficient.

• Block low-value URLs with robots.txt. For faceted navigation you do not need indexed, Google recommends disallowing the filter parameters outright. Its documented example blocks the parameters while allowing an unfiltered listing:

user-agent: Googlebot
disallow: /*?*products=
disallow: /*?*color=
disallow: /*?*size=
allow: /*?products=all$

• Consolidate duplicates. Use rel="canonical" to point filtered or duplicate variants at the version you want crawled, which Google says can decrease crawl volume of the non-canonical versions over time.
• Return the right status codes. Serve 404 or 410 for permanently removed pages. Google treats a 404 as a strong signal not to recrawl, while robots.txt-blocked URLs stay in the crawl queue much longer. And clean up soft 404s, which quietly keep burning budget.
• Make pages efficient to load. Faster responses let Google read more of your site in the same window. Google specifically recommends supporting HTTP caching so that unchanged pages can return 304 (Not Modified), saving your server and its crawlers from re-fetching identical content.
• Do not reach for tools that do not work. Google’s crawlers do not process the non-standard crawl-delay robots.txt rule, and as covered above, noindex is not a crawl-budget lever.

The part most teams skip is measurement. You cannot protect a budget you have never looked at. Google’s Crawl Stats report in Search Console breaks down crawl requests by response code, file type, purpose (discovery versus refresh) and Googlebot type, and it is the fastest way to see whether Google is spending its time on real pages or on junk. On the largest sites, server log files are even more revealing: exporting Googlebot’s actual requests and grouping them by URL pattern usually surfaces the culprit within minutes, whether that is a sort parameter multiplying every category page or a calendar widget generating an endless run of future-dated URLs. The pattern is almost always the same. A small number of URL templates account for the vast majority of wasted crawl, and blocking or consolidating those few templates recovers most of the budget. This kind of log analysis, paired with the discipline in our technical SEO guide, is where crawl budget work actually pays off.

Frequently asked questions

Does crawl budget affect rankings?

Not directly. Google states that crawling is not a ranking factor. A page has to be crawled to appear in results, but increasing your crawl rate will not improve your positions on its own. Crawl budget is about coverage and freshness, not ranking.

How do I know if I actually have a crawl budget problem?

Two signals. First, size and change rate: Google’s guidance targets sites around 1 million+ pages, or 10,000+ pages changing daily. Second, and more telling, a large share of URLs sitting in “Discovered – currently not indexed” in Search Console, combined with a Crawl Stats report showing Google spending its requests on parameter or duplicate URLs rather than your real pages.

Does adding noindex save crawl budget?

Not as a quick fix. Google has to crawl a page to see the noindex tag, so the crawling still happens. Google explicitly says not to use noindex to reallocate crawl budget. Keeping unwanted pages out of the index can indirectly free up crawling over the long term, but to stop crawling entirely you need robots.txt.

Do 404 pages waste crawl budget?

No. Google says pages that serve 4xx status codes, with the exception of 429 (too many requests), do not waste crawl budget. Google attempted the fetch, received a status code and no content, and moves on. A 404 is actually a strong signal to stop recrawling that URL.

Written by Matthis Duarte, a senior SEO and organic growth expert with 10+ years of experience driving organic growth for international brands across highly competitive verticals. He is the founder of Knownful, an independent publication on SEO and organic growth featuring in-depth guides, best practices, playbooks and original analyses, including a free monthly study of which brands AI actually recommends across 10 industries.