How it works
Crawl capacity is how much crawling your server can handle without slowing down. Crawl demand is how much Google wants to crawl, based on popularity and how often pages change.
Google's guide says crawl budget mainly matters for very large sites (1 million+ pages changing weekly), medium sites (10,000+ pages changing daily) or sites with many 'Discovered, currently not indexed' URLs. Every site starts with the same conservative capacity, shared by all Google crawlers [2].
Waste comes from URLs that don't need crawling: Google's 2025 review found faceted navigation caused about half of reported crawl issues and action parameters about a quarter [1].
Why it matters
For most small business sites, crawl budget isn't the problem; poor internal linking or thin content is. For large catalogues, marketplaces, job boards and property portals, it decides how quickly new and updated pages appear in Google. Fixing URL traps, errors and duplicates lets Google spend its crawl on pages that earn money.
By the numbers
pages changing weekly: where crawl budget matters for large sites
Google: Optimize your crawl budget, Jul 2026[2]pages changing daily: the threshold for medium sites
Google: Optimize your crawl budget, Jul 2026[2]of reported crawl issues came from faceted navigation
Google Search Off the Record, episode 103, Feb 2026[1]came from action parameters such as add-to-cart URLs
Google Search Off the Record, episode 103, Feb 2026[1]Key components
Practical example
A Pune property portal had 40,000 listings but Google kept crawling millions of filter combinations (bedrooms × price × area × sort order). New listings took weeks to be indexed. It blocked sort and session parameters in robots.txt, made only key filter pages indexable, fixed slow server responses and kept an accurate sitemap. New listings started appearing in Google within days.
Benefits and limitations
Benefits
- New pages are found and indexed faster.
- Updates to key pages are picked up sooner.
- Less server load from wasted crawling.
Limitations
- Rarely an issue for small sites [2].
- You can't directly set Google's crawl rate upwards.
- Fixes need careful testing to avoid blocking good pages.
Tools and implementation
- Google Search Console: Crawl Stats report.
- Server log analysis: what Googlebot really requests.
- Screaming Frog or Sitebulb: find URL traps and duplicates.
- XML sitemaps with accurate lastmod dates.
Common mistakes
- Worrying about crawl budget on a 50-page site.
- Letting filters create endless crawlable URLs [1].
- Slow or error-prone servers.
- Long redirect chains.
- Using noindex to save crawl budget (Google still has to crawl the page).
Related glossary terms
Our SEO services in Pune diagnose crawl waste on large sites, and our website design and development services fix the URL structures and server issues behind it. Further reading: how to choose an SEO company in Pune.
Get a free consultationFrequently asked questions
Does my site need to worry about crawl budget?
Probably not unless it has hundreds of thousands of pages, very frequent changes or many 'Discovered, currently not indexed' URLs [2].
How do I check my crawl budget?
Use the Crawl Stats report in Search Console and, for large sites, analyse server logs to see what Googlebot requests.
Does noindex save crawl budget?
Not directly. Google has to crawl a page to see the noindex rule. Robots.txt rules or removing URL traps save crawling.
Can I ask Google to crawl more?
Not directly. Faster, more reliable servers and pages people want help Google crawl more.
Sources
- Google Search Off the Record, episode 103 (Feb 2026). Cross-checked with: Search Engine Land.
- Google: Optimize your crawl budget (Jul 2026). Cross-checked with: TechWyse: Google updates crawl budget documentation.
Last updated: 18 November 2026. Every statistic on this page links to its original source and was cross-checked against a second source where one was available.
