Guide

AI bot tracking, from first principles

What is AI bot traffic?

AI bot traffic is every request your website receives from automated agents run by AI companies. It does three different jobs. Some crawlers collect pages for model training. Some build a search index that assistants use to ground and cite their answers. And some fetch a single page in real time because a person just asked ChatGPT, Claude or Perplexity a question that needs it.

Operators increasingly publish a separate user-agent token for each job. OpenAI separates GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User; Perplexity separates PerplexityBot from Perplexity-User. That split is what makes the traffic measurable: once you know which token did what, a log line becomes a statement about training, indexing or a live answer.

Why should you track AI crawlers at all?

Crawler requests are the earliest signal in the chain that ends with a brand being named in an AI answer. An engine cannot cite a page it never fetched. If a firewall rule, a bot-management default or a rate limit answers AI crawlers with 403 or 429, the page drops out of that chain without any visible error in your analytics.

Tracking also keeps the conversation honest. A fetch is not a mention, a citation or a human visit. Teams that report crawler hits as "AI visibility" overstate what happened; teams that ignore crawlers cannot explain why an engine keeps citing a competitor's pricing page instead of theirs. A good tracker keeps these layers apart: requests by bots, citations in answers, and visits by people who clicked through.

Logs or tags: how do bot trackers actually see bots?

There are two ways to observe a request. Server-side sources — web server logs, CDN logs or an edge integration — record every request the server answers, including those from clients that never run JavaScript. Tag-based analytics rely on a script executing in a browser. Many AI crawlers fetch raw HTML and do not execute scripts, so a tag alone usually misses them.

Tags are still useful for the human side: visits that arrive from assistants as referrals, landing pages and conversions. In practice the strongest setups combine both, using logs or CDN data for bots and analytics for people, and label clearly which number came from where.

How do you know a bot is who it claims to be?

A user-agent string is just a header, and anyone can send "GPTBot". Serious analysis therefore verifies the source. Several operators publish the IP ranges their crawlers use, and some support reverse DNS checks. A tracker that matches requests against those published ranges can separate verified crawler traffic from scrapers that borrow a well-known name.

Verification matters most when you act on the data. Blocking or allowing by user agent alone either lets spoofed scrapers through or locks out the real crawler — the opposite of what you wanted.

What should you look for in an AI bot tracker?

We compare tools on the same questions every time. They are the ones that decide whether the numbers will survive a meeting with engineering:

  1. Data source: CDN integration, log upload, log API or a tag — and which hosts and formats are supported.
  2. Bot taxonomy: whether requests are classified by purpose (training, search index, live answer), not only by operator.
  3. Verification: whether IP ranges or reverse DNS are checked before a request is counted as a named bot.
  4. Page and status views: which URLs bots fetch, and how often they hit errors, redirects or blocks.
  5. Connection to outcomes: whether crawler data sits next to citations in AI answers and referral visits.
  6. Retention and price: how long raw data is kept, and which plan unlocks bot analytics.

What can a bot tracker not tell you?

No tracker can see inside a model. A training crawl today does not translate into a mention tomorrow, and a page can be cited from an index that was built weeks ago. Treat crawler data as evidence of access and attention, then measure visibility in the answers themselves. The guides on this site are written with that boundary in mind, and every comparison links the sources it relies on.

Next: compare the tools in our 2026 report on AI bot trackers.