Crawl budget: Google's two size thresholds, and the report that overrules both

Crawl budget is real, and Google's own documentation says most sites should not think about it. There are two size thresholds and one symptom that qualify a site, and if you miss all three the guide says so directly: keep the sitemap current, read the Page Indexing report, and spend the afternoon on something else. This chapter is about how to check which side of that line you are on.

Before you start

You need a rough count of unique URLs on one hostname, an idea of how often those pages change, and access to Search Console for the site. Nothing here needs server logs, although they help. If you have fewer than ten thousand pages and your new posts are picked up within a day or two, you can read the next section and stop.

The three qualifying conditions, and the sentence that dismisses everyone else

Google's crawl budget guidance, read on 2026-08-26, opens by ruling out most of its readers:

"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide."

Then it names who it is for. Three conditions, and one of them is not about size at all.

The third one is the one worth reading twice: "Sites with a large portion of their total URLs classified by Search Console as Discovered - currently not indexed." That is a report you can open right now, and it is the only entry on the list that does not depend on how big you are.

One definition matters before you count. Google treats a site as a hostname: "https://www.example.com/ and https://code.example.com/ are treated as separate sites and have separate crawl budgets." A documentation subdomain and a blog subdomain do not share the pool, so count per hostname rather than per company.

What the budget is made of

Two components, and you can influence them to very different degrees. The documentation splits them as crawl capacity limit and crawl demand, and then says what happens when they disagree: "Even if the crawl capacity limit isn't reached, if crawl demand is low, Google will crawl your site less."

The capacity limit is about your server. Every site starts in the same place — "Every site starts with the same default, conservative crawl capacity limit" — and moves from there based on how the server behaves. Consistent responses and stable latency push it up. Slowdowns, 5xx errors and HTTP 429 push it down.

Demand is about whether Google wants the pages. Three factors are listed: perceived inventory, popularity and staleness. The first carries a note the other two do not: "This is the factor that you can positively control the most." Perceived inventory means the set of URLs Google believes you have, including the duplicates and the parameter variants you never intended to publish.

Crawl budget is not a quota you are given. It is the overlap between what your server can serve and what Google wants to fetch.

How to check whether crawl budget applies to you, in four steps

Four checks, none of which needs a tool you do not already have. Run them in order and stop at the first one that comes back clean, because a single clear negative is enough to close the question.

1. Count unique URLs on the one hostname. Your sitemap total is the fastest estimate; if the sitemap is far smaller than the URL count in Search Console, that gap is itself the finding
2. Ask how many of those change in a normal week. A catalogue of ten thousand pages that changes twice a year is not a rapidly changing site by this definition
3. Open the Page Indexing report and look at the share sitting in Discovered - currently not indexed. A large portion is the third qualifying condition
4. Publish something and watch how long it takes. If it appears the same day, the documentation has already told you the answer

If none of the four turns up anything, stop here. The guide's own fallback for everyone else is two lines long: keep your sitemap up to date, and check the Page Indexing report regularly.

The deliverable: what to fix, and in which order

If you did qualify, the documentation's best-practice list is the work. It is worth reordering by how much of it you control, because the list mixes cheap wins with things that need a rebuild.

The 304 row is the one most sites fail without knowing. Google asks for it explicitly — "Support 304 (Not Modified) HTTP status codes. If a page hasn't changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version" — and a 304 needs a validator on the way out. Our measurement of cache headers on 27 homepages found 13 of them sending neither an ETag nor a Last-Modified, which makes that response impossible no matter what the crawler asks.

Here is the one-line check for your own origin:

# Do you send a validator at all?
curl -sI -A 'Mozilla/5.0' https://example.com/ \
  | grep -i -E 'etag|last-modified|cache-control'

Nothing back means no conditional request can ever succeed. That is a configuration line, not a rebuild.

What going wrong looks like

The first mistake is using robots.txt to shrink perceived inventory. It feels like removal and is not. The documentation is specific about the difference in the context of deleted pages: "Google won't forget a URL that it knows about, but a 404 status code is a strong signal not to crawl that URL again. Blocked URLs, however, will stay part of your crawl queue much longer, and will be recrawled when the block is removed." Blocking is right for pages that exist but should not be fetched. It is the wrong tool for pages that are gone.

The second is the soft 404, which is the most expensive kind of waste because it looks like success. A page that returns 200 with an error message is crawled on the normal schedule forever. Google's own instruction is to eliminate them and check the Page Indexing report for the list, and our survey of what 27 homepages return for a URL that does not exist found this failure on live sites that had no idea.

The third is treating crawl budget as a growth lever. The documentation gives two ways to increase it: add server resources if capacity is the constraint, and improve content quality for the product you are targeting. Neither is a setting. If your pages are crawled and then not indexed, capacity was never the problem, and the answer is in what the pages say.

Where this stops

This chapter cannot tell you that crawl budget is your problem. It can only tell you whether you meet the conditions under which Google says it might be, and for the large majority of sites the honest answer is that it is not. There is no documented number for "how much budget do I have" and no report that shows one, which is why every step above is a check rather than a measurement.

It also does not cover how long any of this takes to show up. Crawling, indexing and ranking run on different clocks, and only one of them has a published figure — see how long it takes to get indexed for the four stages and the point at which waiting stops being reasonable. If what you actually want to know is whether a crawler is reaching your pages at all, that is a different check: see what crawlers receive from a page before assuming the queue is the constraint.

Common questions

Does crawl budget affect rankings?

Not directly. It governs how much of your site gets fetched and how often. A page that is never crawled cannot rank, so the effect exists at the extreme, but on a site whose pages are all being crawled there is nothing left for it to explain.

My pages are crawled but not indexed. Is that a budget problem?

No, and the distinction is the useful part. Crawled means the fetch happened, so capacity and demand both did their job. What follows is the indexing decision, which the crawl budget guide does not govern. The one status that does point back here is Discovered - currently not indexed, where the URL is known and the fetch has not happened.

Will blocking pages in robots.txt give me more budget for the rest?

It can, for pages that exist and are genuinely unimportant — the documentation names infinite scroll variants and re-sorted listings. It will not clear out URLs that no longer exist, because a blocked URL stays in the queue and comes back when the block does.

How do I get more crawl budget?

Two documented routes, and both are outside the markup. Add server resources if the URL inspection tool is reporting Hostload exceeded, and improve content quality, which Google describes for Search as popularity, overall user value, content uniqueness and serving capacity. There is no third route.

Do subdomains share a budget with the main site?

No. Google's crawling infrastructure treats each unique hostname as its own site with its own budget. A docs subdomain that is crawled slowly does not have the main site to blame.