How it works
Googlebot finds URLs through links, sitemaps and previous crawls, then requests them. A page is crawlable when it's linked, returns a 200 status, isn't blocked by robots.txt and loads reliably.
For Search, Googlebot fetches only the first 2MB of an HTML file (measured uncompressed); anything after that cut-off isn't used for indexing. PDFs get 64MB [2].
How much Google crawls depends on your server's health and how much demand there is for your pages. Every site starts on the same conservative crawl capacity, shared by all of Google's crawlers [3].
Why it matters
Most crawl problems are self-inflicted. Google's 2025 review found faceted navigation caused about half of reported crawl issues and action parameters such as add-to-cart links about a quarter [1]. On large catalogue sites, those URL traps can soak up crawling while new products wait to be discovered. For small sites the usual culprits are simpler: orphan pages, JavaScript-only links and staging rules left in robots.txt.
By the numbers
of reported crawl issues came from faceted navigation
Google Search Off the Record, episode 103, Feb 2026[1]came from action parameters such as add-to-cart URLs
Google Search Off the Record, episode 103, Feb 2026[1]of HTML is what Googlebot fetches per page for Search
DebugBear: What Googlebot's 2MB crawl size limit means for SEO, Feb 2026[2]pages changing daily: when crawl budget matters for mid-size sites
Google: Optimize your crawl budget, Jul 2026[3]Key components
Practical example
A Pune B2B manufacturer rebuilt its product range as a JavaScript filter: clicking a category loaded products without changing the URL, so Google could see only the first 12 products. Giving each category and product a real, linked URL, and listing them in the sitemap, made the full range crawlable. Indexed product pages went from 40 to about 300 over the next two months.
Benefits and limitations
Benefits
- New and updated pages are found faster.
- Server resources go to pages that matter.
- Fewer surprises in Search Console's indexing reports.
Limitations
- Crawlable doesn't mean indexed: quality and duplication still decide that.
- Crawl budget only becomes a real constraint on larger sites [3].
- Google decides crawl rate; you can only remove obstacles.
Tools and implementation
- Google Search Console: Crawl Stats and Page indexing reports.
- Screaming Frog or Sitebulb: crawl your site the way a bot does.
- URL Inspection tool: see the HTML Google fetched.
- Server log analysis: what bots actually requested.
Common mistakes
- Links that only work through JavaScript click events.
- Orphan pages with no internal links pointing to them.
- Filters that create millions of crawlable URL combinations [1].
- Very heavy HTML that pushes content past the 2MB cut-off [2].
- Slow or error-prone servers that make Google crawl less.
Related glossary terms
Our SEO services in Pune start with a full crawl of your site, and our website design and development services build crawlable URLs and navigation from the start. Further reading: how to choose an SEO company in Pune.
Get a free consultationFrequently asked questions
What is the difference between crawlability and indexability?
Crawlability is whether Google can reach and fetch a page. Indexability is whether Google is allowed to, and chooses to, store it in its index. A page needs both to rank.
Does Google crawl JavaScript sites?
Yes, Google renders JavaScript, but links and content that exist in the initial HTML are found more reliably. Use real <a href> links for navigation.
Do small sites need to worry about crawl budget?
Rarely. Google's crawl budget guide is aimed at sites with about a million pages changing weekly or 10,000+ changing daily [3]. Small sites should focus on internal links and clean URLs.
How do I check crawlability?
Run a crawler such as Screaming Frog, review Search Console's Crawl Stats and Page indexing reports, and inspect key URLs with the URL Inspection tool.
Sources
- Google Search Off the Record, episode 103 (Feb 2026). Cross-checked with: Search Engine Land.
- DebugBear: What Googlebot's 2MB crawl size limit means for SEO (Feb 2026). Cross-checked with: Search Engine Journal via Pulse: Google updates Googlebot file size limit docs.
- Google: Optimize your crawl budget (Jul 2026). Cross-checked with: TechWyse: Google updates crawl budget documentation.
Last updated: 16 November 2026. Every statistic on this page links to its original source and was cross-checked against a second source where one was available.
