How do AI agents show up in your website traffic?
AI agents can appear in website data in three different ways: as declared fetchers, as browser agents making interactive requests, or as sessions referred by an AI platform.
Latest
Field reports on tracking AI crawler and bot traffic: which tools see GPTBot, ClaudeBot and their peers, and what the numbers mean.
AI crawler analytics measures automated requests made by AI search crawlers, answer engines and training-related bots to a website.
AI agents can appear in website data in three different ways: as declared fetchers, as browser agents making interactive requests, or as sessions referred by an AI platform.
A crawler user-agent string is a claim, not authentication; verify the source against the operator’s bot-specific published IP ranges.
Decide separately whether to allow model training, search indexing and user-triggered retrieval; OpenAI and Anthropic document separate agents for these jobs.
llms.txt is a proposal for a Markdown file placed at a website’s root or another published path.
Crawler data
HTML crawl requests per referred HTML visit, 19–26 June 2025
Anthropic70,900:1
Mistral0.1:1
Crawls per referral, log scale
Grey: comparison count in the same study. Log scale; units differ by study.
Robots.txt tokens grouped by the job each operator documents
Model training 6
GPTBotOpenAIClaudeBotAnthropicGoogle-ExtendedGoogleApplebot-ExtendedApplemeta-externalagentMetaCCBotCommon CrawlSearch index 3
OAI-SearchBotOpenAIClaude-SearchBotAnthropicPerplexityBotPerplexityUser fetch 5
ChatGPT-UserOpenAIClaude-UserAnthropicPerplexity-UserPerplexityMeta-ExternalFetcherMetaDuckAssistBotDuckDuckGoCloudflare AI Crawl Control lets website owners monitor AI crawler activity and create per-crawler allow or block policies for traffic passing through Cloudflare.
AI crawlers are automated agents that fetch web pages for model training, search indexes, AI-generated answers, product discovery, advertising checks, or user-requested browsing.
Search access logs for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot and Googlebot, but treat each match as a candidate rather than proof.
Coming soon
Glossary of AI crawler terms
One email when a report is published or re-checked. No tracking pixels.