How it works
Different bots have different jobs: training crawlers collect content for future models; search crawlers fetch pages to cite in answers; user-triggered agents fetch a page when someone asks about it.
Most identify themselves by user-agent and say they follow robots.txt, so you can allow or block each one. Google-Extended and Applebot-Extended let sites opt out of AI training without leaving Google or Apple search [2].
Since 1 July 2025 Cloudflare blocks known AI crawlers by default on new domains and offers a pay-per-crawl option [2]. If your site uses Cloudflare, check its AI crawler settings.
Why it matters
AI crawlers take far more than they send back. Cloudflare's data for Q3 2026 showed Perplexity fetching about 1,083 pages per referral, Anthropic 988 and OpenAI 309, against about 4.9 for Google [1]. But blocking search-style AI crawlers can also keep you out of AI answers. The decision is a trade-off between protecting content and being visible in AI search, and it can differ by bot.
By the numbers
pages crawled per referral by Perplexity, Q3 2026
Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data), Oct 2026[1]pages crawled per referral by Anthropic, Q3 2026
Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data), Oct 2026[1]Google's crawl-to-referral ratio for comparison
Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data), Oct 2026[1]when Cloudflare began blocking AI crawlers by default
MIT Technology Review: Cloudflare will now block AI bots by default, Jul 2025[2]Key components
Practical example
A Pune ed-tech company saw heavy bot traffic in its server logs and blocked every AI user-agent in robots.txt. A month later it noticed it had vanished from ChatGPT and Perplexity answers about courses it offered. It changed its rules to block training crawlers on its paid course material while allowing AI search crawlers on public marketing pages, and checked that Cloudflare's settings matched. AI mentions returned within weeks.
Benefits and limitations
Benefits
- Control over how your content is used.
- Reduced server load from aggressive bots.
- Allowing search crawlers supports AI visibility.
Limitations
- Not every bot follows robots.txt.
- Blocking can remove you from AI answers.
- Crawler names and purposes keep changing.
Tools and implementation
- robots.txt: per-bot allow and disallow rules.
- Cloudflare AI Crawl Control or similar CDN settings.
- Server logs: which bots visit and how often.
- AI visibility tools: whether blocking affected mentions.
Common mistakes
- Blocking all AI bots without considering AI search visibility.
- Blocking Googlebot when you meant to block Google-Extended.
- Forgetting CDN settings that override your robots.txt intentions [2].
- Never checking server logs.
- Assuming robots.txt stops every scraper.
Related glossary terms
Our SEO services in Pune set an AI crawler policy that balances protection and visibility, and our website design and development services apply it across robots.txt, server and CDN. Further reading: how to choose an SEO company in Pune.
Get a free consultationFrequently asked questions
Should I block AI crawlers?
It depends on your goals. Blocking training crawlers protects content; blocking AI search crawlers can remove you from AI answers. Many businesses allow search crawlers on public pages and restrict training crawlers.
Does blocking Google-Extended affect my Google rankings?
Google-Extended controls whether your content is used for Gemini training, separately from Googlebot and Search [2]. Blocking it isn't the same as blocking Google Search.
How much traffic do AI crawlers send back?
Very little compared with what they take. In Q3 2026 Perplexity crawled about 1,083 pages per referral and Anthropic 988, versus about 4.9 for Google [1].
Is my site blocking AI crawlers without me knowing?
Possibly. Cloudflare has blocked known AI crawlers by default on new domains since July 2025 [2]. Check your CDN settings as well as robots.txt.
Sources
- Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data) (Oct 2026). Cross-checked with: SEOmator: GEO data report, Oct 2026 (Cloudflare Radar).
- MIT Technology Review: Cloudflare will now block AI bots by default (Jul 2025). Cross-checked with: The Register: Cloudflare AI crawler changes and Google-Extended.
Last updated: 18 November 2026. Every statistic on this page links to its original source and was cross-checked against a second source where one was available.
