How it works
After crawling, Google checks the page's status code, meta robots and X-Robots-Tag directives, and canonical signals. A noindex directive or a non-200 status keeps the page out.
If several URLs show the same content, Google groups them and indexes one. It uses around 40 signals to choose that canonical [2], and treats your rel=canonical as a strong hint rather than a rule [1].
Robots.txt doesn't control indexing: a blocked URL can still be indexed from links, and Google can't see a noindex tag on a page it isn't allowed to crawl [3].
Why it matters
Duplication is everywhere: Google's Gary Illyes has said about 60% of the internet is duplicate [5], so Google is constantly deciding which version to keep. If your signals conflict (a canonical pointing one way, internal links and the sitemap pointing another), Google may index the wrong URL or neither. 'Crawled, currently not indexed' in Search Console usually means Google saw the page and decided it wasn't worth storing.
By the numbers
of the internet is duplicate, according to Google
Search Engine Roundtable: Google, 60% of the internet is duplicate, Nov 2022[5]signals Google uses to choose a canonical URL
Search Engine Journal: Google uses about 40 signals to determine canonical URLs, Dec 2024[2]of HTML is fetched for indexing; content beyond it is ignored
DebugBear: What Googlebot's 2MB crawl size limit means for SEO, Feb 2026[4]Key components
Practical example
A Pune coaching institute had 600 near-identical 'course in [area]' pages. Search Console showed most as 'Duplicate, Google chose different canonical'. Merging them into 12 strong course pages with genuine local detail, redirecting the rest and updating internal links resolved the clash, and the course pages started ranking for area searches they'd never appeared for.
Benefits and limitations
Benefits
- Clear signals mean Google indexes the URL you want.
- Keeps thin and utility pages out of results.
- Concentrates links and relevance on one version of each page.
Limitations
- Google decides what to index; you can only make the case.
- Canonicals are hints, so Google can overrule them [1].
- Indexing doesn't guarantee rankings or traffic.
Tools and implementation
- Google Search Console: Page indexing report and URL Inspection.
- Screaming Frog: audit status codes, noindex and canonicals at scale.
- site: searches: a rough spot check, not a count.
- Browser extensions that show meta robots and canonical tags.
Common mistakes
- Blocking a page in robots.txt and expecting noindex to work [3].
- Canonical tags pointing to redirected or noindexed URLs.
- Sitemaps full of URLs you've told Google not to index.
- Leaving a site-wide noindex on after launch.
- Publishing hundreds of near-identical location pages.
Related glossary terms
Fixing indexing problems is often the fastest win in our SEO services in Pune, because pages that aren't indexed can't rank at all. Further reading: how to choose an SEO company in Pune.
Get a free consultationFrequently asked questions
Why is my page not indexed?
Common reasons are a noindex tag, a canonical pointing elsewhere, duplication with another URL, thin content, or the page being new and not yet crawled. Search Console's URL Inspection tool shows the reason Google gives.
How long does indexing take?
It varies from hours to weeks. Internal links from strong pages and an up-to-date sitemap help Google find new pages sooner.
Can I force Google to index a page?
No. You can request indexing in Search Console, but Google still decides. Improving the page and its internal links is what changes the outcome.
Does robots.txt stop indexing?
No. It stops crawling. To keep a page out of results, allow crawling and add a noindex tag [3].
Sources
- Google Search Central: How to specify a canonical URL (Current doc). Cross-checked with: Ahrefs: Google uses about 40 canonicalization signals.
- Search Engine Journal: Google uses about 40 signals to determine canonical URLs (Dec 2024). Cross-checked with: Search Engine Roundtable.
- Google Search Central: Introduction to robots.txt (Current doc). Cross-checked with: Google: How Google interprets the robots.txt specification.
- DebugBear: What Googlebot's 2MB crawl size limit means for SEO (Feb 2026). Cross-checked with: Search Engine Journal via Pulse: Google updates Googlebot file size limit docs.
- Search Engine Roundtable: Google, 60% of the internet is duplicate (Nov 2022). Cross-checked with: Digital Information World.
Last updated: 16 November 2026. Every statistic on this page links to its original source and was cross-checked against a second source where one was available.
