How it works
Crawlers fetch yoursite.com/robots.txt before crawling. The file groups rules by user-agent (the crawler's name) with allow and disallow lines for URL paths, and can point to your XML sitemap.
Google supports four fields: user-agent, allow, disallow and sitemap. It doesn't support crawl-delay, and it ignores anything beyond the first 500 KiB of the file [2].
Blocking a URL stops Google fetching it, but the URL can still appear in results if other pages link to it. To keep a page out of Google, use a noindex tag (on a crawlable page) or password protection [2].
Why it matters
One wrong line in robots.txt can hide a whole site from search, which makes it one of the first files we check in any audit. Used well, it stops crawlers wasting time on filter, search and cart URLs: Google's own crawl data shows faceted navigation caused about half of the crawl issues reported to it in 2025 [1]. It's also where you decide how AI crawlers may use your site.
By the numbers
of robots.txt is all Google reads; the rest is ignored
Google Search Central: Introduction to robots.txt[2]of reported crawl issues in 2025 came from faceted navigation
Google Search Off the Record, episode 103, Feb 2026[1]of HTML per page is what Googlebot fetches for indexing
DebugBear: What Googlebot's 2MB crawl size limit means for SEO, Feb 2026[3]Key components
Practical example
A Pune furniture store saw thousands of URLs like /sofas?colour=grey&sort=price in Search Console. Adding Disallow rules for the sort and filter parameters, while leaving category pages crawlable and canonicalised, cut wasted crawling. Separately, a 'Disallow: /' line left over from the staging site had been blocking the new blog for a month; removing it was the single biggest fix.
Benefits and limitations
Benefits
- Keeps crawlers away from low-value or endless URL spaces.
- Protects server resources on large or slow sites.
- Lets you set separate rules for different crawlers, including AI bots.
Limitations
- Doesn't remove pages from Google or keep them private [2].
- Polite crawlers follow it; bad bots ignore it.
- Blocked pages can't pass on what Google would learn from crawling them, including noindex tags.
Tools and implementation
- Google Search Console robots.txt report: fetch status and errors.
- Screaming Frog: test which URLs your rules block.
- Server logs: see which bots actually request what.
- Your CMS or SEO plugin: edit the file safely.
Common mistakes
- Copying 'Disallow: /' from staging to the live site.
- Blocking a page you want deindexed, so Google never sees its noindex tag.
- Blocking CSS or JavaScript that Google needs to render pages.
- Relying on crawl-delay, which Google doesn't support [2].
- Putting private URLs in robots.txt, which publicly advertises them.
Related glossary terms
We check robots.txt on every audit in our SEO services in Pune, and our website design and development services include a launch checklist so staging rules never reach the live site. Further reading: how to choose an SEO company in Pune.
Get a free consultationFrequently asked questions
Will robots.txt remove a page from Google?
No. It stops crawling, not indexing. A blocked page can still show in results if it's linked from elsewhere. Use a noindex tag or password protection instead [2].
Where does robots.txt go?
At the root of each host, for example https://www.example.com/robots.txt. Subdomains need their own file.
Does Google support crawl-delay?
No. Google supports user-agent, allow, disallow and sitemap [2]. If Googlebot is overloading your server, fix server capacity or contact Google through Search Console.
Should I block AI crawlers in robots.txt?
It depends on whether you want your content used by AI tools. Most AI companies publish their crawler names, so you can allow or block them separately from Googlebot.
Sources
- Google Search Off the Record, episode 103 (Feb 2026). Cross-checked with: Search Engine Land.
- Google Search Central: Introduction to robots.txt (Current doc). Cross-checked with: Google: How Google interprets the robots.txt specification.
- DebugBear: What Googlebot's 2MB crawl size limit means for SEO (Feb 2026). Cross-checked with: Search Engine Journal via Pulse: Google updates Googlebot file size limit docs.
Last updated: 17 November 2026. Every statistic on this page links to its original source and was cross-checked against a second source where one was available.
