Most small and medium sites never need to think about crawl budget. Large or fast-changing sites do. Here is how Google decides how much to crawl and how to stop wasting it.
This guide explains crawl budget using Google’s documentation and what I see on client sites. It is part of my technical SEO service.
Crawl budget is the number of URLs Googlebot can and wants to crawl on your site. It depends on crawl capacity (how much your server can handle) and crawl demand (how much Google wants your content). Google says it mainly matters for very large sites, sites with fast-changing content, and sites with many pages stuck in “Discovered, currently not indexed”.
| Site | Care about crawl budget? |
|---|---|
| 1 million+ unique pages changing about weekly | Yes |
| 10,000+ unique pages changing daily | Yes |
| Many URLs stuck in “Discovered, currently not indexed” | Yes, investigate |
| Small or medium site, new pages crawled the same day | Usually no |
Google notes these numbers are rough estimates, not exact thresholds.
In Search Console, open Settings, then Crawl stats. Watch total crawl requests, average response time and the share of responses by type (OK, redirect, not found, server error). A rising response time or many errors are the first things to fix.
An online store with 4,000 products generates 300,000 filter URLs through colour, size and sort parameters. Googlebot spends most of its time on those, while new products wait days to be crawled. Blocking sort and filter parameters, keeping a few useful filtered category pages indexable, and cleaning the sitemap lets Google reach new products much faster.
I audit and fix speed, Core Web Vitals, indexing, redirects and schema, and send you a clear report of every change.
The amount of crawling Google can and wants to do on your site, based on crawl capacity and crawl demand.
Usually not. Google says crawl budget mainly matters for very large or fast-changing sites.
Blocking useless URL patterns in robots.txt stops Google spending crawls on them. Do not block pages you want indexed.
Yes. Google must crawl a page to see the noindex rule.
Use the Crawl stats report in Search Console settings.
Facts checked against these official pages on 10 October 2026. Read my editorial policy.
A technical SEO audit finds what stops Google from crawling, indexing and understanding your pages. This is the 6-step, 30-check process I use before any other SEO work.
Read article
Schema markup tells search engines exactly what a page is about: an article, a product, a business, a person. Here is which types matter, how to add them and the mistakes to avoid.
Read article
Core Web Vitals measure how fast your main content loads, how quickly the page reacts and how stable it is. Here are the thresholds and the fixes that work on real sites.
Read articleSend your website and I will reply with a free audit.