Updated
Which AI crawlers exist and what are their user agents?
AI crawlers are automated agents that fetch web pages for model training, search indexes, AI-generated answers, product discovery, advertising checks, or user-requested browsing.

Summarize with
TL;DR
- GPTBot crawls content that may be used to train OpenAI foundation models, while OAI-SearchBot supports ChatGPT Search and ChatGPT-User handles user-initiated visits.[1]
- ClaudeBot is Anthropic’s model-training crawler, and Claude-SearchBot and Claude-User serve distinct search and user-requested access purposes.[5]
- Google-Extended is a robots.txt control token rather than a separate HTTP user agent.[2]
- PerplexityBot, CCBot, Amazonbot and DuckAssistBot each have documented robots.txt tokens, but their purposes and published IP information differ.[8][7]
- Robots.txt is a policy signal rather than a guaranteed block, so crawler verification should combine user-agent, IP, DNS and request-behavior checks.[5]
AI crawlers are automated agents that fetch web pages for model training, search indexes, AI-generated answers, product discovery, advertising checks, or user-requested browsing. The most clearly documented agents include OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic’s ClaudeBot; PerplexityBot; Google’s Google-Extended robots token; Applebot-Extended; Meta-ExternalAgent; CCBot; Amazonbot; DuckAssistBot; and xAI’s Grok Bot.[1]
This reference separates three things that are often confused:
- An HTTP user agent is the identifier sent in a request header.
- A robots.txt token is the name used in a
User-agentdirective. - An IP range helps verify that a request actually came from the claimed operator.
A robots.txt token is not always a separate HTTP crawler. Google-Extended, for example, controls how Google-crawled content may be used for Gemini but does not identify a separate Google request agent.[2] Applebot-Extended is documented as a secondary user agent for content-use controls rather than as a separate published crawler network.[3]
AI crawler user agents at a glance
The table below lists the main AI-related crawlers and control tokens documented in the research sources. User-agent strings can change, so use each operator’s current documentation or IP file when implementing verification rules.
| Operator | HTTP user agent or robots token | Main purpose | robots.txt token | Published IP information |
|---|---|---|---|---|
| OpenAI | GPTBot/1.4 |
Crawls content that may be used to train OpenAI foundation models.[1] | GPTBot |
OpenAI publishes a GPTBot IP list.[4] |
| OpenAI | OAI-SearchBot/1.4 |
Surfaces websites in ChatGPT Search.[1] | OAI-SearchBot |
OpenAI publishes a SearchBot IP list.[1] |
| OpenAI | ChatGPT-User/1.0 |
Makes user-initiated visits from ChatGPT.[1] | ChatGPT-User |
OpenAI publishes a ChatGPT-User IP list.[1] |
| OpenAI | OAI-AdsBot/1.0 |
Checks landing pages submitted for ChatGPT advertisements.[1] | OAI-AdsBot |
OpenAI publishes an AdsBot IP list.[1] |
| Anthropic | ClaudeBot |
Collects public web content that could contribute to model training.[5] | ClaudeBot |
Anthropic says it does not publish crawler IP ranges because it uses service-provider public IPs.[5] |
| Anthropic | Claude-SearchBot |
Crawls to improve search-result quality.[5] | Claude-SearchBot |
No separate published range is identified in the cited documentation. |
| Anthropic | Claude-User |
Supports user-initiated access.[5] | Claude-User |
No separate published range is identified in the cited documentation. |
| Perplexity | Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
Crawls and indexes web content.[6] | PerplexityBot |
Perplexity publishes a PerplexityBot list.[6][7] |
| Perplexity | Perplexity-User |
Supports user actions in Perplexity.[7] | Perplexity-User |
Do not assume the PerplexityBot list applies to this agent.[8][7] |
| Existing Google crawler user agents | Crawls pages used by Google services.[2] | Google-Extended |
No separate Google-Extended IP range exists because it is not a separate HTTP request identity.[2] | |
| Apple | Applebot | Crawls publicly available web content.[9] | Applebot |
Apple publishes an Applebot CIDR JSON file.[3] |
| Apple | Applebot-Extended | Provides controls over how content is used, including model-training opt-out.[3] | Applebot-Extended |
A separate Applebot-Extended IP list is not identified.[3] |
| Meta | meta-externalagent/1.1 |
Crawls for foundation-model training and product indexing.[10] | meta-externalagent |
No IP range is published in the cited documentation.[10] |
| Meta | Meta-ExternalFetcher | Fetches individual links at a user’s request, including agentic tasks.[10] | Meta-ExternalFetcher |
No IP range is published in the cited documentation.[10] |
| ByteDance | Bytespider |
Listed by Cloudflare as an AI crawler.[4] | Bytespider |
No first-party IP list was located for this reference.[4] |
| Common Crawl | CCBot/2.0 (+https://commoncrawl.org/faq/) |
Collects web data for Common Crawl’s open crawl repository.[11] | CCBot |
Common Crawl publishes a CCBot IP JSON file.[11] |
| Amazon | Amazonbot/0.1 |
Improves Amazon products and services and may support Amazon AI model training.[12][5] | Amazonbot |
Amazon publishes Amazonbot IP addresses.[13][12] |
| DuckDuckGo | DuckAssistBot/1.2 |
Fetches pages for AI-assisted DuckDuckGo answers that cite sources.[14] | DuckAssistBot |
DuckDuckGo publishes addresses and a JSON file.[14] |
| xAI | Grok Bot | The cited documentation provides security information, but this reference does not confirm a complete user-agent string or purpose.[10] | Not confirmed here | Not confirmed here. |
What does each major AI crawler do?
OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot
OpenAI documents several distinct agents rather than one universal crawler. GPTBot is associated with crawling content that may be used to train OpenAI foundation models, while OAI-SearchBot supports ChatGPT Search.[1]
ChatGPT-User is used for user-initiated visits, so it should not automatically be treated as a background training crawler. OAI-AdsBot checks landing pages submitted as ChatGPT advertisements, and OpenAI states that data collected by that bot is not used to train foundation models.[1]
The corresponding robots.txt tokens are GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. OpenAI treats GPTBot and OAI-SearchBot controls independently, so blocking one does not automatically block the other.[1]
Anthropic: ClaudeBot, Claude-SearchBot and Claude-User
ClaudeBot collects public web content that could contribute to model training. Claude-SearchBot crawls to improve search-result quality, while Claude-User supports access initiated by a Claude user.[5]
Anthropic documents the tokens ClaudeBot, Claude-SearchBot and Claude-User. Its guidance includes a Crawl-delay: 1 example for ClaudeBot and recommends applying rules to every relevant subdomain.[5]
Anthropic says it does not publish crawler IP ranges because its crawler uses public IP addresses supplied by service providers.[5] That makes user-agent and reverse-DNS-only identification less conclusive than a published operator-owned range.
PerplexityBot and Perplexity-User
PerplexityBot is used for crawling and indexing, while Perplexity-User supports user actions in Perplexity.[7] Perplexity’s documented example begins as follows:
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity says PerplexityBot follows robots.txt. If text access is disallowed, it may still index a domain, headline and brief factual summary.[6] The documented PerplexityBot IP list should not automatically be applied to Perplexity-User because the cited material does not confirm that both use the same ranges.[1][8]
Google-Extended
Google-Extended is commonly described as an AI crawler, but technically it is a robots.txt control token rather than a standalone HTTP user agent. Google uses its existing crawler user agents, and Google-Extended controls whether content crawled by Google may be used to train future Gemini models and for grounding.[2]
A rule such as User-agent: Google-Extended therefore does not identify a separate request source. It is a policy control layered onto Google’s established crawling infrastructure.[2]
Applebot and Applebot-Extended
Applebot crawls publicly available web content, and Apple says it follows standard robots.txt directives.[9] Apple publishes an Applebot CIDR JSON file for identifying Applebot traffic.[3]
Applebot-Extended is documented as a secondary user agent that gives publishers controls over content use, including opting out of training Apple’s foundation models.[3] Apple does not identify a separate Applebot-Extended IP list in the cited documentation.
Meta-ExternalAgent and Meta-ExternalFetcher
Meta-ExternalAgent, shown in Meta’s documentation with the prefix meta-externalagent/1.1, crawls for foundation-model training and product indexing.[10] Meta-ExternalFetcher fetches individual links at a user’s request, including for agentic tasks.[10]
Meta identifies meta-externalagent and Meta-ExternalFetcher as the relevant robots.txt tokens. The cited Meta documentation does not provide IP ranges for either crawler, so server operators should not treat a matching name alone as proof of origin.[8]
Bytespider
Cloudflare lists Bytespider as an AI crawler associated with ByteDance.[4] The research did not locate ByteDance first-party documentation confirming a fuller user-agent string, purpose, robots token behavior or IP list.[4][1]
A 2025 academic measurement study reported that Bytespider ignored robots.txt directives in the researchers’ tests.[4] That result means a Disallow rule should not be treated as a guaranteed technical block for this crawler.
CCBot
CCBot is Common Crawl’s crawler. Common Crawl describes its work as maintaining an open repository of web crawl data and gives CCBot as the robots.txt token for opting out.[11]
Common Crawl publishes a CCBot IP JSON file and recommends verifying requests using published ranges and reverse DNS because other crawlers may falsely identify themselves as CCBot.[11]
Amazonbot
Amazonbot helps Amazon improve its products and services, provide more accurate customer information and potentially train Amazon AI models.[12] Amazon documents a user-agent example containing Amazonbot/0.1 and says the bot honors allow and disallow directives in robots.txt.[13][12]
Amazon fetches the host-level robots.txt file or uses a cached copy from the previous 30 days. If it cannot fetch the file, Amazon says it behaves as though the file does not exist.[12] Amazon publishes a separate Amazonbot IP-address page.[13]
DuckAssistBot
DuckAssistBot crawls pages in real time for AI-assisted DuckDuckGo answers that cite their sources.[14] The documented user agent is:
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
DuckDuckGo uses DuckAssistBot as the robots.txt token. A disallow takes effect after 72 hours and does not affect organic search rankings or inclusion in search results.[14]
How should robots.txt rules be written?
Robots.txt can express the operator’s requested access policy, but it is not an authentication mechanism and cannot force every crawler to comply. The academic study found that most large AI companies tested respected robots.txt, but some applications and crawlers did not; the study specifically reported noncompliance by Bytespider.[1][4]
Use separate rules when the purposes of two agents differ. For example, a site might allow a search crawler while disallowing a training crawler, subject to each operator’s documented behavior.
Example robots.txt file
- Open the site’s
robots.txtfile at the root of the host. - Add the operator-specific tokens you intend to control.
- Use
Disallow: /to request that the named agent access no paths. - Use an empty
Disallow:to allow access. - Test the file and monitor server logs after publishing it.
# Allow ChatGPT Search but request no training crawl
User-agent: OAI-SearchBot
Disallow:
User-agent: GPTBot
Disallow: /
# Request that Claude's training crawler does not fetch the site
User-agent: ClaudeBot
Disallow: /
# Request that PerplexityBot does not fetch page text
User-agent: PerplexityBot
Disallow: /
# Control Google's Gemini training and grounding use
User-agent: Google-Extended
Disallow: /
# Request no Apple model-training use
User-agent: Applebot-Extended
Disallow: /
# Request no Meta foundation-model or product-indexing crawl
User-agent: meta-externalagent
Disallow: /
# Opt out of Common Crawl
User-agent: CCBot
Disallow: /
# Request no Amazonbot access
User-agent: Amazonbot
Disallow: /
# Request no DuckDuckGo AI answer crawl
User-agent: DuckAssistBot
Disallow: /
This example expresses separate requests for each documented token; it does not prove that every request will stop. Operators should also review each vendor’s current policy because crawler names, purposes and enforcement practices can change.[8]
How can you verify an AI crawler’s identity?
A user-agent string is self-declared and can be spoofed. Common Crawl explicitly warns that other crawlers may falsely identify themselves as CCBot and recommends checking reverse DNS and published IP ranges.[11]
A practical verification process is:
- Record the complete request user-agent, source IP, timestamp, requested URL and response status.
- Compare the source IP with the operator’s current published JSON or CIDR list.
- Perform reverse-DNS verification where the operator documents an official hostname pattern.
- Forward-confirm the resulting hostname back to the same IP address.
- Check request behavior, including rate, path selection, robots.txt retrieval and repeated visits.
- Treat traffic as unverified if the user agent matches but the network origin does not.
This process is strongest for operators that publish IP ranges, including OpenAI, Perplexity, Apple, Common Crawl, Amazon and DuckDuckGo. It is less conclusive for Anthropic, which says it uses service-provider public IPs rather than publishing crawler ranges.[5][8]
What is the difference between blocking a user agent and blocking an IP range?
Blocking a user agent uses a request’s self-declared User-Agent header and is easy to configure in robots.txt or a web server. Blocking an IP range operates at the network layer and can be stronger when the range is current and belongs to the claimed operator, but it can also block legitimate traffic if ranges change or are shared. Neither method alone is a universal guarantee against spoofing or distributed crawling.[5]
What are the main limitations of AI crawler lists?
AI crawler references become outdated quickly because operators can add agents, change user-agent versions, move infrastructure or introduce new access modes. A list should therefore show the date checked and link to the operator’s current documentation rather than treating a name as permanent.
The most important limitations are:
- A robots token may not correspond to a separate HTTP crawler, as with Google-Extended.[2]
- User-initiated agents can have a different purpose from automated crawlers, as with ChatGPT-User, Claude-User and Perplexity-User.
- A robots.txt rule is a policy signal, not a technical access barrier.[8]
- A user-agent string can be forged.
- Some operators publish IP ranges, while others do not.[8][5]
- A crawler’s stated purpose may include several uses, such as search, indexing, product improvement or model training.[10][1]
- A block can affect discoverability or AI-generated answers differently depending on the operator’s product behavior.[14]
For ongoing monitoring, combine robots.txt review with access logs, CDN bot controls, published IP lists and alerts for new user-agent strings. Allowing a crawler also doesn’t guarantee it can read your pages: Finseo’s own audit of AI crawler readability found that most AI crawlers never run JavaScript, and across 20,293 audited pages on 524 sites, 8.9% returned nothing readable.
Frequently asked questions
What is an AI crawler user agent?
An AI crawler user agent is the identifier an automated AI-related web client sends in an HTTP request. It can indicate that the client is fetching content for model training, search, indexing, product improvement, advertising checks or a user-requested action.[10][2]
What is the GPTBot user agent?
GPTBot is OpenAI’s documented crawler for content that may be used to train OpenAI foundation models.[1] A documented example uses `GPTBot/1.4`, and the corresponding robots.txt token is `GPTBot`.[1]
What is the ClaudeBot user agent?
ClaudeBot is Anthropic’s crawler for collecting public web content that could contribute to model training.[5] The robots.txt token is `ClaudeBot`; Anthropic also documents separate Claude-SearchBot and Claude-User agents for search and user-initiated access.[5]
Is Google-Extended a crawler?
Google-Extended is not a separate HTTP crawler user agent. It is a robots.txt control token that governs how content crawled by Google may be used for Gemini model training and grounding.[2]
Does robots.txt block AI crawlers?
Robots.txt communicates an access preference to compliant crawlers, but it is not a guaranteed technical block. Academic testing found that most large AI companies studied respected robots.txt, while some crawlers, including Bytespider in that study, did not.[10][4]
How can a website verify an AI crawler?
Compare the complete user-agent string with the operator’s documentation, then verify the source IP against a current published range where available. Reverse DNS and forward-confirmation checks add evidence, but a matching user-agent alone is not sufficient because crawlers can impersonate one another.[3][8]
Sources
- [1]Overview of OpenAI Crawlers — developers.openai.com
- [2]Google's common crawlers | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers — developers.google.com
- [3]About Applebot - Apple Support — support.apple.com
- [4]Bot reference — developers.cloudflare.com
- [5]Does Anthropic crawl data from the web, and how can site owners block the crawler? | Claude Help Center — anthropic.com
- [6]How does Perplexity follow robots.txt? — perplexity.ai
- [7]Perplexity Crawlers — docs.perplexity.ai
- [8]Grok Bot security | SpaceXSI Docs — docs.x.ai
- [9]Applebot model training and individual privacy rights - Apple Support — support.apple.com
- [10]Meta Web Crawlers | Meta for Developers — developers.facebook.com
- [11]Common Crawl - CCBot — commoncrawl.org
- [12]About AmazonBot — developer.amazon.com
- [13]Amazon Web Search crawlers | Amazonbot — developer.amazon.com
- [14]Is DuckAssistBot related to DuckDuckGo? - DuckDuckGo Help Pages — duckduckgo.com