How it works
Every request to your server is logged with the URL, time, status code and user-agent. Filtering for verified bots shows real crawler behaviour, not estimates.
Analysis reveals which sections get crawled most, where bots hit errors or redirects, which important pages are rarely visited and how much crawling goes to parameter URLs.
It also shows AI crawler activity. Cloudflare's Q3 2026 data put Perplexity at about 1,083 pages crawled per referral and Anthropic at 988, against about 4.9 for Google [3].
Why it matters
Search Console shows samples and summaries; logs show everything. On large sites, logs prove where crawl budget is wasted: Google found faceted navigation behind about half of reported crawl issues [1]. They also confirm whether Googlebot fetches the full HTML within its 2MB limit and how AI bots are using your content, which helps you set a sensible crawler policy.
By the numbers
of reported crawl issues came from faceted navigation
Google Search Off the Record, episode 103, Feb 2026[1]of HTML per page is what Googlebot fetches for indexing
DebugBear: What Googlebot's 2MB crawl size limit means for SEO, Feb 2026[2]pages crawled per referral by Perplexity, Q3 2026
Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data), Oct 2026[3]Google's crawl-to-referral ratio for comparison
Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data), Oct 2026[3]Key components
Practical example
A Pune online marketplace believed Google was crawling its new product pages. Log analysis showed 70% of Googlebot requests went to sort and filter URLs, while new products waited days for a first crawl, and that an AI training crawler was hitting the site thousands of times an hour. It blocked the filter parameters, rate-limited the AI crawler, and new products began being crawled within hours.
Benefits and limitations
Benefits
- Shows real crawler behaviour, not estimates.
- Finds crawl waste and hidden errors.
- Reveals AI bot activity on your site.
Limitations
- Needs log access, which some hosts restrict.
- Large files need specialist tools.
- Most valuable on larger sites [1].
Tools and implementation
- Screaming Frog Log File Analyser.
- Botify, JetOctopus or Oncrawl for large sites.
- CDN logs and analytics (such as Cloudflare).
- Google Search Console Crawl Stats for comparison.
Common mistakes
- Trusting user-agent strings without verifying bots.
- Analysing a single day instead of weeks.
- Looking at logs without a clear question.
- Ignoring CDN logs when the CDN serves most requests.
- Not acting on what the analysis finds.
Related glossary terms
Our SEO services in Pune include log analysis for larger sites, and our website design and development services help you get log access and fix what it reveals. Further reading: how to choose an SEO company in Pune.
Get a free consultationFrequently asked questions
Do I need log file analysis for a small site?
Usually not. Search Console's Crawl Stats is enough for most small sites. Logs pay off on large or complex sites.
How do I get my server logs?
From your hosting control panel, server or CDN. Some shared hosts limit access; ask your provider.
How do I know a bot is really Googlebot?
Verify with a reverse DNS lookup; fake bots often copy Googlebot's user-agent.
Can logs show AI crawlers?
Yes. They show requests from GPTBot, ClaudeBot, PerplexityBot and others, which helps you decide what to allow [3].
Sources
- Google Search Off the Record, episode 103 (Feb 2026). Cross-checked with: Search Engine Land.
- DebugBear: What Googlebot's 2MB crawl size limit means for SEO (Feb 2026). Cross-checked with: Search Engine Journal via Pulse: Google updates Googlebot file size limit docs.
- Technology Checker: bot traffic statistics Q3 2026 (Cloudflare Radar data) (Oct 2026). Cross-checked with: SEOmator: GEO data report, Oct 2026 (Cloudflare Radar).
Last updated: 18 November 2026. Every statistic on this page links to its original source and was cross-checked against a second source where one was available.
