What Are AI Crawlers?

AI crawlers explained: which bots AI companies use, how much they take versus send back, Cloudflare's default blocking, and what to allow in robots.txt.

Updated: 18 November 2026Also known as: AI bots, LLM crawlers, GPTBot and ClaudeBot4 min read

Get Free Consultation

Speak with our Digital Marketing Expert today.

Quick definitionAI crawlers are bots run by AI companies to fetch web pages, either to train models or to find current pages for AI answers. Examples include OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot and Google-Extended.

How it works

Different bots have different jobs: training crawlers collect content for future models; search crawlers fetch pages to cite in answers; user-triggered agents fetch a page when someone asks about it.

Most identify themselves by user-agent and say they follow robots.txt, so you can allow or block each one. Google-Extended and Applebot-Extended let sites opt out of AI training without leaving Google or Apple search [2].

Since 1 July 2025 Cloudflare blocks known AI crawlers by default on new domains and offers a pay-per-crawl option [2]. If your site uses Cloudflare, check its AI crawler settings.

Why it matters

AI crawlers take far more than they send back. Cloudflare's data for Q3 2026 showed Perplexity fetching about 1,083 pages per referral, Anthropic 988 and OpenAI 309, against about 4.9 for Google [1]. But blocking search-style AI crawlers can also keep you out of AI answers. The decision is a trade-off between protecting content and being visible in AI search, and it can differ by bot.

By the numbers

1,083:1

pages crawled per referral by Perplexity, Q3 2026

Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data), Oct 2026[1]
988:1

pages crawled per referral by Anthropic, Q3 2026

Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data), Oct 2026[1]
4.9:1

Google's crawl-to-referral ratio for comparison

Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data), Oct 2026[1]
1 Jul 2025

when Cloudflare began blocking AI crawlers by default

MIT Technology Review: Cloudflare will now block AI bots by default, Jul 2025[2]

Key components

Training crawlersCollect content for model training, such as GPTBot.
Search crawlersFetch pages to cite in AI answers, such as OAI-SearchBot.
User agentsFetch a page when a user asks, such as ChatGPT-User.
Control tokensGoogle-Extended and Applebot-Extended for training opt-outs.
Network controlsCDN and firewall settings such as Cloudflare's.

Practical example

A Pune ed-tech company saw heavy bot traffic in its server logs and blocked every AI user-agent in robots.txt. A month later it noticed it had vanished from ChatGPT and Perplexity answers about courses it offered. It changed its rules to block training crawlers on its paid course material while allowing AI search crawlers on public marketing pages, and checked that Cloudflare's settings matched. AI mentions returned within weeks.

Benefits and limitations

Benefits

  • Control over how your content is used.
  • Reduced server load from aggressive bots.
  • Allowing search crawlers supports AI visibility.

Limitations

  • Not every bot follows robots.txt.
  • Blocking can remove you from AI answers.
  • Crawler names and purposes keep changing.

Tools and implementation

Common mistakes

Our SEO services in Pune set an AI crawler policy that balances protection and visibility, and our website design and development services apply it across robots.txt, server and CDN. Further reading: how to choose an SEO company in Pune.

Get a free consultation

Frequently asked questions

Should I block AI crawlers?

It depends on your goals. Blocking training crawlers protects content; blocking AI search crawlers can remove you from AI answers. Many businesses allow search crawlers on public pages and restrict training crawlers.

Does blocking Google-Extended affect my Google rankings?

Google-Extended controls whether your content is used for Gemini training, separately from Googlebot and Search [2]. Blocking it isn't the same as blocking Google Search.

How much traffic do AI crawlers send back?

Very little compared with what they take. In Q3 2026 Perplexity crawled about 1,083 pages per referral and Anthropic 988, versus about 4.9 for Google [1].

Is my site blocking AI crawlers without me knowing?

Possibly. Cloudflare has blocked known AI crawlers by default on new domains since July 2025 [2]. Check your CDN settings as well as robots.txt.

Sources

  1. Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data) (Oct 2026). Cross-checked with: SEOmator: GEO data report, Oct 2026 (Cloudflare Radar).
  2. MIT Technology Review: Cloudflare will now block AI bots by default (Jul 2025). Cross-checked with: The Register: Cloudflare AI crawler changes and Google-Extended.

Last updated: 18 November 2026. Every statistic on this page links to its original source and was cross-checked against a second source where one was available.