Bot Trackers

Field reports on tracking AI crawler and bot traffic: which tools see GPTBot, ClaudeBot and their peers, and what the numbers mean.

Which tools track AI crawler and bot traffic?

AI crawler analytics measures automated requests made by AI search crawlers, answer engines and training-related bots to a website.

Diagram of a crawler request passing CDN, origin server and access logs, with the trackers that read each layer

How do AI agents show up in your website traffic?

AI agents can appear in website data in three different ways: as declared fetchers, as browser agents making interactive requests, or as sessions referred by an AI platform.

Should you block AI crawlers or let them in?

Decide separately whether to allow model training, search indexing and user-triggered retrieval; OpenAI and Anthropic document separate agents for these jobs.

Crawler data

What the published numbers say

Crawl-to-referral ratio

HTML crawl requests per referred HTML visit, 19–26 June 2025

Anthropic70,900:1

Mistral0.1:1

Crawls per referral, log scale

Requests for /llms.txt in three log studies

Grey: comparison count in the same study. Log scale; units differ by study.

Dejan.ai

1,150robots.txt requests

4llms.txt requests

OtterlyAI

62,100AI-bot visits

84llms.txt requests

N. Hall

1,549announced /ai/llms.txt

2unannounced /llms.txt

AI crawler directory

Robots.txt tokens grouped by the job each operator documents

Model training 6

  • GPTBotOpenAI
  • ClaudeBotAnthropic
  • Google-ExtendedGoogle
  • Applebot-ExtendedApple
  • meta-externalagentMeta
  • CCBotCommon Crawl

Search index 3

  • OAI-SearchBotOpenAI
  • Claude-SearchBotAnthropic
  • PerplexityBotPerplexity

User fetch 5

  • ChatGPT-UserOpenAI
  • Claude-UserAnthropic
  • Perplexity-UserPerplexity
  • Meta-ExternalFetcherMeta
  • DuckAssistBotDuckDuckGo

What does Cloudflare's AI Crawl Control do?

Cloudflare AI Crawl Control lets website owners monitor AI crawler activity and create per-crawler allow or block policies for traffic passing through Cloudflare.

Which AI crawlers exist and what are their user agents?

AI crawlers are automated agents that fetch web pages for model training, search indexes, AI-generated answers, product discovery, advertising checks, or user-requested browsing.

How do you find AI bot traffic in your server logs?

Search access logs for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot and Googlebot, but treat each match as a candidate rather than proof.

Coming soon

Glossary of AI crawler terms

Get new field reports by email

One email when a report is published or re-checked. No tracking pixels.