Updated

Should you block AI crawlers or let them in?

Decide separately whether to allow model training, search indexing and user-triggered retrieval; OpenAI and Anthropic document separate agents for these jobs.

13 minute readMarkdown versionFact-checked

Illustration: three bots at a fence turnstile, one let through, one carrying pages stopped by a barrier, one waiting

TL;DR

  • Decide separately whether to allow model training, search indexing and user-triggered retrieval; OpenAI and Anthropic document separate agents for these jobs.[6][1]
  • Block GPTBot while allowing OAI-SearchBot when the goal is to exclude OpenAI training collection without automatically giving up ChatGPT search eligibility.[5]
  • Treat robots.txt as a preference signal, not a security boundary; use WAF or CDN enforcement when the page must not be delivered.[6]
  • Google-Extended does not control ordinary Google Search inclusion or ranking, while Google’s Search generative-AI control separately addresses selected AI Search features.[16][17]
  • Charging is a possible access policy, but Cloudflare documents Pay Per Crawl as a closed beta rather than guaranteed demand or revenue.[7]

There is no universal “block” or “allow” answer. Site owners should decide crawler by crawler and separate three jobs: model training, search indexing, and user-triggered retrieval. Then choose whether to allow, disallow, enforce a block, or charge for each job.

The practical default for many sites is selective access: allow search crawlers when AI discovery and referrals matter, disallow training-specific collection when model-development rights are unacceptable, and use WAF or CDN rules when a provider must not fetch the content at all. Robots.txt expresses a preference, but it is not an access-control mechanism.[1]

This framework uses the following decision sequence:

  1. Identify what the crawler is doing.
  2. Decide whether that use is acceptable.
  3. Select a purpose-specific robots.txt rule.
  4. Add edge enforcement if delivery must be prevented.
  5. Consider paid access only where a real commercial arrangement exists.
  6. Measure the result and review the policy on a fixed date.

What are AI crawlers actually doing?

“AI crawler” describes several different activities rather than one uniform type of traffic.

  • Training crawlers collect material that may be used to train or improve foundation models. GPTBot and ClaudeBot are examples.[1][2]
  • Search crawlers discover and index pages for AI-powered search products. OAI-SearchBot, Claude-SearchBot and PerplexityBot are examples.
  • User-triggered fetchers retrieve a page because a person asked an assistant to read or use it. ChatGPT-User, Claude-User and Perplexity-User serve this role.[3][4]

Blocking one job does not necessarily block the others. OpenAI says GPTBot and OAI-SearchBot are controlled independently, so a site can disallow GPTBot while allowing OAI-SearchBot to support ChatGPT search visibility.[5] Anthropic similarly documents separate agents for training, search and user-directed access.[1][6]

The distinction matters because the costs differ:

  • Blocking training collection can protect against unwanted model-development use while preserving discovery.
  • Blocking search indexing can reduce eligibility for AI search answers and links.
  • Blocking user-triggered retrieval can stop an assistant from reading a page for a specific request.
  • Blocking traffic at the network edge can prevent access regardless of whether a crawler respects robots.txt.

A permission signal and a delivery decision are therefore separate. A robots.txt rule tells a cooperating crawler what the site prefers; it does not authenticate the request or guarantee that content will not be fetched.[4]

What should you block, allow or charge?

Crawler job Example tokens What blocking it costs What blocking it protects Mechanism
Model training GPTBot, ClaudeBot, Google-Extended, Applebot-Extended Possible loss of future model-training participation and any indirect benefits associated with it.[7] Keeps content outside the specified training collection or use.[8] Purpose-specific robots.txt rules; edge blocking where access itself is unacceptable.[1][9]
Search indexing OAI-SearchBot, Claude-SearchBot, PerplexityBot Lower visibility in the relevant AI search product and potentially fewer links or referrals.[10] Reduces discovery and indexing by that product.[11] robots.txt and, where necessary, verified WAF/CDN rules.[12][13]
User-triggered fetches ChatGPT-User, Claude-User, Perplexity-User Users may receive answers that cannot use the blocked page as a source.[1] Prevents direct retrieval of the page for assistant requests.[1][7] Provider-specific robots controls, plus network enforcement when required.[14]
Google AI presentation and grounding Google-Extended; Google Search controls Less or no inclusion in selected AI-generated search experiences or grounding contexts.[9][15] Separates AI presentation, grounding and training decisions from ordinary Search.[10][13] Search Console generative-AI control, robots.txt and Google product tokens.[16][17]
Paid retrieval Crawler-specific Cloudflare actions Possible loss of unpaid access or crawls that do not accept the price.[7] Converts eligible content access into a paid request.[18][19] CDN or WAF policy with Allow, Block or Charge actions.[18]

Allow when the crawler’s purpose supports discovery, referrals, user service or another measurable objective. Disallow when a provider offers a purpose-specific opt-out and the site wants to preserve other access. Block when the site must prevent delivery rather than merely state a preference. Charge only when the site has access to an operational paid-crawl system and is willing to reject crawlers that do not pay.

Cloudflare’s AI Crawl Control documents all three per-crawler actions—Allow, Charge and Block.[18][19] Pay Per Crawl is described as a closed beta, and successful retrievals can be charged at a site-defined price.[20][19] That makes charging an available access option, not proof that a crawler will pay or that the resulting income will offset lost referrals.

How do the main crawler tokens differ?

OpenAI: GPTBot is not OAI-SearchBot

OpenAI says GPTBot crawls content that may be used to train its foundation models. Disallowing GPTBot indicates that the site’s content should not be used for that training purpose.[5]

OAI-SearchBot is used to surface websites in ChatGPT search and is not used to crawl content for foundation-model training.[5] Blocking it can remove the site from ChatGPT search answers, although OpenAI says navigational links may remain.[5]

ChatGPT-User is different again: it supports user actions in ChatGPT and Custom GPTs, including requests to visit a web page. OpenAI says this agent is not used for automatic web crawling or generative-AI training, and its robots.txt treatment may not apply to every user-initiated action.[4]

Decision: If the goal is “block AI training but keep ChatGPT search,” target GPTBot rather than blocking OpenAI traffic as a whole.

Anthropic: training, search and user access are separate

Anthropic documents ClaudeBot for material that may contribute to training, Claude-SearchBot for search indexing and relevance, and Claude-User for user-directed page access.[4]

Anthropic says its bots honor robots.txt, but each subdomain should be configured because a rule on one host does not automatically express the same preference for every other host.[4]

Decision: Use separate rules for ClaudeBot, Claude-SearchBot and Claude-User when the site wants different treatment for training, search or user requests.

Perplexity: search and user retrieval are not training collection

PerplexityBot supports Perplexity search and helps surface and link websites. Perplexity recommends allowing it for search inclusion.[3] Perplexity-User handles user-requested retrieval and is not described as an automatic crawler or foundation-model training collector.

Perplexity says Perplexity-User generally ignores robots.txt because the fetch is user-requested.[3] A network-level denial may therefore be necessary if the site must prevent that page from being retrieved.

Decision: A robots.txt rule may express a preference for PerplexityBot, but it should not be treated as a guaranteed block for Perplexity-User.

Google: Google-Extended is not a firewall identity

Google-Extended is a robots.txt product token controlling how content collected by Google’s infrastructure may be used for specified Gemini Apps and Vertex AI services, including future model training and grounding.[16] Google says Google-Extended does not affect inclusion or ranking in Google Search.[16][17]

Google-Extended is not a separate HTTP crawler that can simply be denied through a firewall match.[21] Google’s Search generative-AI control is a separate Search Console setting. Google says the control can include or exclude a property from links, content and grounding in AI Overviews, AI Mode and other listed generative-AI features, while other Search ranking and inclusion signals remain unaffected.[17]

Decision: Do not use Google-Extended as a synonym for “remove my site from Google AI answers.” Specify whether the objective is model training, Gemini grounding, AI Search presentation or ordinary Search.

Apple: Applebot-Extended controls use, not crawling

Applebot-Extended is a robots.txt purpose token for excluding content from training Apple’s general-purpose foundation models.[22] Apple says it does not itself crawl websites. Pages can remain available for Apple search while the site uses the token to express a training preference.[23][6]

Apple’s documentation also distinguishes other controls, including noindex and nosnippet, for search appearance and descriptions.[22]

Decision: Use Applebot-Extended when the concern is training use, and use search-specific directives when the concern is indexing or displayed content.

What does robots.txt actually guarantee?

Robots Exclusion Protocol guidance is useful but limited. RFC 9309 defines rules for crawler access preferences and how user-agent groups, Allow and Disallow records are interpreted.[24][25] It explicitly states that robots.txt is not an authorization mechanism.[1]

A robots.txt file therefore cannot:

  • Authenticate a crawler.
  • Stop a non-cooperating client from requesting a page.
  • Guarantee that a provider has deleted information already collected.
  • Guarantee citations, rankings, referrals or revenue.
  • Distinguish every downstream use unless the provider documents that distinction.
  • Replace passwords, authentication, rate limits or WAF controls.

Robots.txt also makes listed paths publicly discoverable, so it should not be used to conceal sensitive URLs.[24]

Why is robots.txt advisory, and how do WAF rules differ?

Robots.txt is a published preference file that compliant crawlers may read before fetching content. It is not a security boundary, and RFC 9309 says it does not replace access authorization.[24]

A WAF or CDN rule acts during the request itself. It can match a user-agent, verified bot identity, IP range, path, rate or other request property and return a denial response. That provides stronger enforcement, but it introduces its own risks: user-agent strings can be forged, shared infrastructure can create false positives, and blocking a mixed-use crawler can remove both unwanted and wanted access.

Use robots.txt to communicate purpose-specific preferences. Use authentication, origin controls, WAF rules or CDN policies when the business requirement is “this request must not receive the page.”

A purpose-specific robots.txt policy can disallow training-oriented agents while allowing search-oriented agents. The following is an example, not a universal policy. Provider behavior and product scopes should be checked before deployment.

# Example only: block selected training collection,
# while allowing selected search and user-fetch agents.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Explicitly allow selected search crawlers.
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Explicitly allow selected user-triggered fetchers.
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

This file does not create a universal “no AI training” guarantee. Google-Extended and Applebot-Extended are product tokens rather than separate crawlers, and user-triggered agents have provider-specific behavior.[13][16] A provider may also use other agents or apply separate contractual and technical controls.

If the site needs a hard block, pair the published preference with edge enforcement. Verify crawler identity where possible, monitor false positives and avoid assuming that one user-agent string represents only one downstream purpose.

What are the six steps for an AI crawler policy?

1. Inventory the traffic

List every relevant crawler, token, IP range where published, request path and observed frequency. Separate training, search, user-triggered retrieval and mixed-use traffic rather than grouping all AI requests under one label.[23]

2. Define the permitted purpose

For each crawler, record whether the site allows model training, search discovery, AI-answer presentation, grounding, user-triggered retrieval or none of these. Google’s controls demonstrate why “AI use” is too broad a category for a reliable policy.[6]

3. Assess the cost of exclusion

Estimate what blocking could remove: search eligibility, citations, referrals, direct assistant retrieval, licensing opportunities or future model participation. Do not convert crawler volume into business value automatically; crawl-to-referral ratios combine different crawler jobs and can be distorted by missing referral headers.[2]

4. Choose the control layer

Use a purpose-specific robots.txt rule for a cooperative provider. Use Search Console or provider-specific settings where available. Add WAF or CDN enforcement when preventing page delivery is the requirement.[3]

5. Decide whether access should be paid

Set a price only if the site has a defined payment workflow, acceptable terms and a way to reconcile successful deliveries. Cloudflare’s Pay Per Crawl documentation describes a minimum price of $0.001 per crawl and a payment-intent workflow, while its product remains in closed beta.[26]

6. Measure, document and review

Track requests, bytes, status codes, search appearances, AI referrals, conversions, licensing outcomes, errors and false positives. Record the crawler, purpose, path scope, mechanism, expected consequence and review date. Revisit the policy when a provider changes its crawler, search product or compliance behavior.

Should you use Cloudflare’s controls?

Cloudflare’s AI Crawl Control allows site owners to manage individual crawlers with Allow, Charge or Block actions.[13][23] Its monitoring features can also compare crawler activity with robots.txt compliance.[2]

The important distinction is between preference and enforcement:

  • A robots.txt policy communicates the site’s intended use.
  • A Cloudflare block enforces denial at the edge.
  • A Cloudflare charge requests payment for successful content access.
  • A provider’s own policy determines whether it honors the published preference.

Cloudflare’s Pay Per Crawl workflow can return HTTP 402 Payment Required with pricing when a crawler has not supplied payment intent, then provide access after the crawler presents the required payment headers.[19] Certain infrastructure paths, including /robots.txt and /sitemap.xml, remain free to crawl.[27]

Use edge controls when the site has a firm access requirement. Do not assume that enabling a block for a mixed-use crawler affects only training. Cloudflare’s own documentation distinguishes search, training and agent policies, and the result depends on the configured crawler and other rules.[23][5]

Is charging AI crawlers better than blocking them?

Charging can be attractive for publishers that want to make content available without offering unlimited free retrieval. Cloudflare documents a per-crawl pricing model in which the site owner sets a price and successful content retrieval is charged.[19][26]

But charging is not the same as guaranteed revenue. The product is documented as a closed beta, and a crawler must support the payment workflow or receive a payment-required response instead of the page.[20][18] There is no basis for assuming that every AI provider will participate or that paid crawls will compensate for lost referrals.

Cloudflare also documents a separate Pay Per Use beta in which buyers define downstream uses and prices.[7] That is different from charging for a successful page retrieval. Keep these fields separate:

  • Price per crawl.
  • Permitted downstream use.
  • Buyer participation.
  • Payment status.
  • Referral or conversion value.
  • Contractual restrictions.

A site should choose Charge only after deciding what access is being sold and what happens when the crawler declines to pay.

What does the research say about blocking?

Research does not support a single web-wide percentage of sites that block AI crawlers. Studies use different populations, languages, dates and units of measurement.[1]

One study found that more than 10% of consistently popular domains explicitly disallowed AI crawlers after major agent announcements, but its sample was based on domains that remained in the Tranco Top 100K across a defined period.[1] Another study reported restrictions affecting more than 5% of tokens across several training corpora and more than 25% of tokens from critical or head domains; those are token-weighted findings, not percentages of websites.[8]

A study of 4,900 domains across 14 languages reported different opt-out rates across language-resource tiers, while a selected news-site study found a large difference between reputable and misinformation-site samples.[28] These findings show that publisher category, popularity, language and corpus importance affect the result.

Actual access can also diverge from declarations. A large empirical study analyzed approximately 3.9 million requests across 36 websites and found that crawler compliance varied by bot category and directive strictness.[29] The operational lesson is simple: measure observed requests separately from robots.txt policy state.

How should site owners measure the decision?

Start with a dated baseline before changing access. Record:

  • Requests by crawler and user-agent.
  • Verified versus unverified bot traffic.
  • HTML requests, bytes and response codes.
  • Search appearances and indexed pages.
  • AI answer citations and links where measurable.
  • Referral sessions and conversions.
  • Revenue or leads attributed to AI referrals.
  • Crawl errors and false-positive blocks.
  • Paid retrievals, prices and payment failures.

Cloudflare defines crawl-to-referral ratios using HTML crawl requests divided by HTML requests referred by the platform, with referrals identified through the Referer header.[21] Cloudflare warns that native-app referrals may lack a Referer header, which can overstate the ratio.[10][21] The same blind spot affects conversion reporting: Finseo’s own attribution study found that analytics records about 1 in 10 AI-driven conversions, and it describes a post-purchase survey that closes the gap.

Published July 2025 figures illustrate why dates matter: Cloudflare reported 38,065.7 Anthropic crawl requests per referral, 1,091.4 for OpenAI and 194.8 for Perplexity in that period.[2] These are historical platform-level measurements, not a forecast for an individual site and not a training-only comparison.[1]

The right test is not “did the site block AI?” It is “did the chosen policy improve the site’s defined outcome without causing unacceptable loss elsewhere?”

Frequently asked questions

Should I block AI crawlers?

Block only the crawler jobs that conflict with the site’s objectives. A common selective policy is to disallow training-specific collection, allow search crawlers when discovery matters and use network controls for user-triggered fetches that must not receive the page.[1]

Should I block GPTBot or not?

Block GPTBot if the site does not want its content used for OpenAI foundation-model training.[21][1] OpenAI documents GPTBot separately from OAI-SearchBot, so blocking GPTBot does not require blocking ChatGPT search discovery.[5]

Can I allow AI crawlers through robots.txt?

Yes, robots.txt can express an `Allow` preference for a named user agent, but it does not guarantee access, citations or compliance.[4] A provider may use separate agents for search, training and user requests, so each relevant token should be evaluated separately.[21]

Can I block AI training bots while keeping ordinary search?

Often, yes. GPTBot, ClaudeBot, Google-Extended and Applebot-Extended have purpose-specific controls that can be separated from selected search mechanisms.[1] Google’s Search generative-AI control is also separate from ordinary Search ranking and inclusion signals.[16]

Does blocking Google-Extended remove a site from Google Search?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search.[16][17] It is a robots.txt product token covering specified Gemini-related training and grounding uses, so Google Search and Gemini-use decisions should not be treated as identical.[7]

Is robots.txt enough to stop AI scraping?

No. RFC 9309 says robots.txt is not access authorization.[24] If preventing delivery is essential, use authentication or an appropriately configured WAF/CDN rule, while recognizing that bot identification and mixed-use traffic can create enforcement errors.

Sources

  1. [1]Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers — arxiv.org
  2. [2]The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals — blog.cloudflare.com
  3. [3]Perplexity Crawlers — docs.perplexity.ai
  4. [4]Does Anthropic crawl data from the web, and how can site owners block the crawler? | Anthropic Privacy Center — privacy.claude.com
  5. [5]Overview of OpenAI Crawlers — developers.openai.com
  6. [6]Have it both ways: stay discoverable in search while disallowing AI training — blog.cloudflare.com
  7. [7]Pay Per Use: when AI uses your work, you should get paid — blog.cloudflare.com
  8. [8]Consent in Crisis: The Rapid Decline of the AI Data Commons — proceedings.neurips.cc
  9. [9]robots.txt setting — developers.cloudflare.com
  10. [10]The crawl before the fall… of referrals: understanding AI’s impact on content providers — blog.cloudflare.com
  11. [11]New opportunities, control and insights for website owners — blog.google
  12. [12]AI Features and Your Website | Google Search Central  |  Documentation  |  Google for Developers — developers.google.com
  13. [13]Your site, your rules: new AI traffic options for all customers — blog.cloudflare.com
  14. [14]AI Preferences (aipref) — datatracker.ietf.org
  15. [15]draft-ietf-aipref-vocab-08 — datatracker.ietf.org
  16. [16]Google's common crawlers | Google Crawling Infrastructure  |  Crawling infrastructure  |  Google for Developers — developers.google.com
  17. [17]Search generative AI control - Search Console Help — support.google.com
  18. [18]Manage AI crawlers — developers.cloudflare.com
  19. [19]Introducing pay per crawl: Enabling content owners to charge AI crawlers for access — blog.cloudflare.com
  20. [20]What is Pay Per Crawl? — developers.cloudflare.com
  21. [21]Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content — blog.cloudflare.com
  22. [22]About Applebot - Apple Support — support.apple.com
  23. [23]Say it once: introducing Bot Preference Sync — blog.cloudflare.com
  24. [24]RFC 9309: Robots Exclusion Protocol — rfc-editor.org
  25. [25]draft-ietf-aipref-attach-05 - Associating AI Usage Preferences with Content in HTTP — datatracker.ietf.org
  26. [26]Set a pay per crawl price — developers.cloudflare.com
  27. [27]Pay Per Crawl FAQ — developers.cloudflare.com
  28. [28]Whose Robots.txt? Consent-Infrastructure Inequality Across the Languages of Web Corpora — aclanthology.org
  29. [29]Scrapers selectively respect robots.txt directives:evidence from a large-scale empirical study — arxiv.org

Related topics

ComparisonsGuideCrawler policy

Get new field reports by email

One email when a report is published or re-checked. No tracking pixels.

More from Bot Trackers