How it works
Crawl: Googlebot discovers URLs through links and XML sitemaps, while robots.txt tells it which paths not to fetch.
Render: Google processes HTML, CSS and JavaScript to see the page as a user would. Content that only appears after complex scripts can be missed or delayed.
Index: Google decides whether to store the page, choosing one 'canonical' version among duplicates. Noindex tags, canonical tags and status codes (200, 301, 404, 410, 5xx) all steer this decision.
Since 5 July 2024 Google crawls every site with its smartphone crawler: if content isn't accessible on mobile, it can't be indexed [3].
Why it matters
Great content can't rank if it isn't indexed. Technical problems are also invisible to most business owners: a site can look fine in a browser while blocking crawlers, serving duplicate URLs or loading slowly on 4G phones. Speed is measurable and widely failed: only 48% of mobile origins passed all three Core Web Vitals in July 2025 [4]. The good news for small and mid-sized sites: Google's crawl-budget guidance is aimed at sites with 1 million+ pages changing weekly or 10,000+ pages changing daily [2], so most business sites should focus on indexing hygiene and speed rather than crawl budget.
By the numbers
date from which Google crawls all sites with Googlebot Smartphone
Google Search Central blog, Jun 2024[3]mobile / desktop origins passing all Core Web Vitals (Jul 2025)
HTTP Archive Web Almanac 2025: Performance, 2025[4]page counts at which Google says crawl budget becomes a concern
Google: Optimize your crawl budget, Doc updated Jul 2026[2]of web traffic in India comes from mobile devices
StatCounter: Desktop vs Mobile vs Tablet, India, Jul 2026[1]Key components
Practical example
A Pune manufacturer relaunches its WordPress site. Three weeks later, Search Console shows pages dropping out of the index. The cause: the developer left 'Discourage search engines from indexing this site' ticked from staging, which adds a noindex directive to every page. Unticking it, resubmitting the XML sitemap and requesting indexing for key pages via URL Inspection resolves it: but only because someone was monitoring the Page indexing report. A technical SEO checklist at launch would have caught it on day one.
Benefits and limitations
Benefits
- Removes invisible barriers that cap all other SEO work.
- Faster pages improve conversions as well as rankings.
- Fixes are often one-off and site-wide, so impact scales.
- Clean architecture makes future content easier to rank.
Limitations
- Technical fixes enable rankings but don't create relevance or authority on their own.
- Some issues need developer time and testing, especially on custom or JavaScript-heavy sites.
- Google says a perfect performance score purely for SEO isn't the best use of effort [5].
Tools and implementation
- Google Search Console: Page indexing, Crawl stats, Core Web Vitals and URL Inspection.
- Screaming Frog / Sitebulb: full-site crawls.
- PageSpeed Insights and Lighthouse: lab and field speed data.
- Rich Results Test and Schema Markup Validator.
- Server log files: show exactly what Googlebot requests.
Common mistakes
- Launching with noindex or a robots.txt 'Disallow: /' left over from staging.
- Hiding content or links on the mobile version.
- Redirect chains and links pointing to redirected URLs after a migration.
- Returning a 200 status for 'page not found' pages (soft 404s).
- Obsessing over crawl budget on a 200-page site.
Related glossary terms
Technical audits are the first step of our SEO services in Pune. If your site needs rebuilding rather than fixing, our website design and development services team handles that.
Get a free consultationFrequently asked questions
What's the difference between crawling and indexing?
Crawling is Google fetching a page; indexing is Google deciding to store it so it can appear in results. A page can be crawled but not indexed.
Do small websites need to worry about crawl budget?
Usually not. Google's guidance targets sites with 1 million+ pages changing weekly or 10,000+ pages changing daily [2].
How often should a technical SEO audit be done?
A full audit at launch or redesign, then a lighter monthly check of Search Console's indexing and Core Web Vitals reports is a sensible baseline.
Sources
- StatCounter: Desktop vs Mobile vs Tablet, India (Jul 2026). Cross-checked with: StatCounter June 2026 view: 67.15% mobile.
- Google: Optimize your crawl budget (Doc updated Jul 2026). Cross-checked with: Search Engine Roundtable.
- Google Search Central blog (Jun 2024). Cross-checked with: Search Engine Roundtable.
- HTTP Archive Web Almanac 2025: Performance (2025). Cross-checked with: Christoph Olivier Consulting stats summary.
- Google: Understanding page experience (Mar 2024). Cross-checked with: Search Engine Roundtable.
Last updated: 16 November 2026. Every statistic on this page links to its original source and was cross-checked against a second source where one was available.
