# What does Cloudflare's AI Crawl Control do?

> Cloudflare AI Crawl Control lets website owners monitor AI crawler activity and create per-crawler allow or block policies for traffic passing through Cloudflare.

Source: https://bestaibottrackers.com/cloudflare-ai-crawl-control · Publisher: Bot Trackers · Updated: 2026-11-13

## TL;DR
- Cloudflare AI Crawl Control monitors and allows or blocks identified AI crawlers on traffic routed through Cloudflare; it is not a universal internet-wide block.[13]
- Cloudflare’s public sources conflict about default AI crawler policies, so inspect each zone’s current Search, Agents, and Training settings.[4][7]
- Pay Per Crawl uses payment intent, HTTP 200 responses, and HTTP 402 Payment Required responses, but Cloudflare still describes it as a private or closed beta.[5]
- Content Signals communicate preferences for search, AI input, and training but do not technically block access.[3][11]
- Cloudflare’s crawl-to-referral ratios are request-based, can be skewed by missing referrers, and should not be treated as November 2026 performance readings.[19]

## What does Cloudflare’s AI Crawl Control do?

Cloudflare AI Crawl Control lets website owners monitor AI crawler activity and create per-crawler allow or block policies for traffic passing through Cloudflare.[1] It is not a universal internet-wide block: the domain must proxy traffic through Cloudflare before the product can inspect or control requests.[2]

The most useful way to understand Cloudflare’s AI crawler tools is to separate four functions:

1. **Content Signals** communicate whether content may be used for search, AI input, or model training.[3]
2. **AI Crawl Control** monitors crawler activity and applies allow or block actions to Cloudflare-routed requests.[4]
3. **Pay Per Crawl** proposes a payment flow for successful crawler requests, but remains in beta.[5]
4. **Analytics** measure crawler requests and some referral activity, but do not provide a complete measure of AI-attributed business value.[6]

Cloudflare’s current documentation also contains conflicting descriptions of default policies. A developer policy page updated on 2 November 2026 says new domains allow Search, Agents, and Training by default, while a September 2026 press release describes different defaults for ad-supported sites.[7] Check the actual settings for each zone instead of assuming that Cloudflare blocks AI bots automatically.




## How does AI Crawl Control work?

AI Crawl Control, formerly called AI Audit, gives Cloudflare customers a dashboard for observing AI service requests and managing access.[1] Its controls include crawler activity, request patterns, individual crawler allow or block policies, and robots.txt compliance monitoring.[1]

The product identifies crawlers through user-agent-based detection on all plans. Enterprise plans with Bot Management can also use more thorough Bot Management detection IDs and configurable analytics timeframes.[8] Cloudflare says those detection IDs are verified, whereas user-agent matching can be spoofed.[9]

The crawler view can show:

- Allowed requests
- Unsuccessful requests
- Request trends
- Robots.txt violations
- The configured action
- Crawler and operator details
- Hostnames and paths
- Status codes

An unsuccessful request does not necessarily mean that AI Crawl Control blocked it. Another Cloudflare rule, an origin failure, or a response error can also produce an unsuccessful request.[2]

Cloudflare’s plan distinction matters when interpreting the data. All plans have crawler detection and allow/block controls, but free-plan analytics cover no more than 24 hours. Enterprise plans with Bot Management can configure longer analytics timeframes.[8]

## What are the default AI bot settings?

Cloudflare’s current public documentation does not support a simple statement such as “Cloudflare blocks AI bots by default.”

A developer policy page updated on 2 November 2026 says that new domains default to **Allow** for Search, Agents, and Training.[4] For existing zones, the page says owners choose actions in Security settings rather than giving one fixed default.[4]

The available actions vary by category:

- **Search:** Allow, Block, or Block on pages with ads.[4][7]
- **Agents:** Allow, Block, or Block on pages with ads.[4][7]
- **Training:** Allow, Disallow, or Block on pages with ads.[4][7]

However, Cloudflare’s rollout materials describe different settings. A 1 July 2026 changelog said new domains would block Training and block Agents on pages displaying ads while keeping Search allowed.[10] A 15 September 2026 press release described ad-supported sites as allowing Search, disallowing Training, and blocking Agents on pages carrying ads; it said all three were allowed for other sites.[7]

These sources conflict with the later developer policy page. The safest operational rule is to inspect each zone’s current Security settings, document the selected action, and test representative URLs rather than relying on a general default.

## What does “block AI bots” mean in Cloudflare?

Cloudflare’s block controls apply to identified crawler requests that reach a site through Cloudflare.[7] They do not stop every AI service from accessing content through other routes, and they cannot control traffic that bypasses Cloudflare.

Blocking also depends on detection. A request using a misleading or spoofed user agent may not be classified in the same way as a verified crawler.[9] Enterprise Bot Management detection IDs provide an additional detection layer, but the research report does not establish that they identify every automated request.

A block policy is stronger than a robots.txt preference. Cloudflare’s robots.txt documentation says compliance is voluntary, and some operators may ignore directives.[3][11] A 2025 empirical study similarly found that bots varied considerably in robots.txt compliance, with evidence of user-agent spoofing and known bots that rarely or never checked robots.txt.[12]

## How does Pay Per Crawl work?

Pay Per Crawl is Cloudflare’s proposed mechanism for charging AI crawlers for access to content. A crawler can send payment intent in request headers; for a successful HTTP 200 response, the request can proceed, while Cloudflare may return HTTP 402 Payment Required with pricing when payment is required.[5]

Cloudflare says it acts as the Merchant of Record and provides the technical infrastructure for the transaction flow.[5] WAF or Bot Management rules take precedence, so a crawler blocked by another rule cannot reach the zone through the charge action.[7]

The published rules include several important limits:

- The same configured price applies to all crawlers assigned to Charge.[13]
- Each page recrawl incurs the configured charge.[13]
- Only successful HTTP responses are billed.[13]
- `/robots.txt`, `/sitemap.xml`, `/security.txt`, `/.well-known/security.txt`, and `/crawlers.json` are always free.[13][14]
- Site owners can exempt URI patterns.[14]
- An origin `crawler-price` response header can support dynamic pricing.[15][14]

Cloudflare’s documentation uses `USD 3.14` as an example value, not as a universal rate.[14] The cited pages do not publish a standard Cloudflare price, public settlement schedule, general availability date, or requirement that every crawler pay.[14][13]

The feature is still described as beta: the overview calls it a private beta, while the feature page calls it a closed beta.[2] Pay Per Crawl should therefore be treated as an experimental monetization option rather than a generally available way to fund every AI crawler visit.

## What are Content Signals?

Content Signals are declarations about permitted use rather than technical access controls. The three categories distinguish:

- **Search:** using content to build a search index and provide search results, excluding AI-generated search summaries.[16]
- **AI input:** using content as input to AI models, including retrieval-augmented generation, grounding, and real-time generative answers.[3]
- **AI training:** using content to train or fine-tune models.[17][4]

A sample policy can allow search while declining AI input and training:

```text
User-Agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no
Allow: /
```

The signal communicates a publisher preference. It does not technically prevent an operator from requesting the page.[18][2] Cloudflare directs site owners to AI Crawl Control when they need an enforcement action rather than a declaration.[19]

This makes Content Signals useful for expressing a nuanced policy. A publisher can remain discoverable in conventional search while stating that content should not be used as AI input or training data. Whether a crawler honors that preference remains a separate question.

## What analytics does Cloudflare provide?

AI Crawl Control can break crawler data down by date, crawler, operator, status code, hostname, and path.[8] The Crawlers tab also reports allowed and unsuccessful requests and robots.txt violations.[2]

Cloudflare’s GraphQL API exposes dashboard data for custom reporting. Available filters include real client requests, user agent, status, URL path, referrer host, and Bot Management detection IDs.[9] Referrer-domain filtering is limited to paid plans.[2][9]

The analytics should not be interpreted as a complete measure of AI visibility, content value, or revenue. Finseo's own research reviews six studies on this gap and reports that analytics records only [about 1 in 10 AI-driven conversions](https://www.finseo.ai/blog/ai-search-attribution). Cloudflare’s crawl/referral ratio compares HTML requests from relevant AI or search user agents with HTML requests whose `Referer` header contains a hostname associated with that platform.[18]

Cloudflare reported the following June 2025 examples:

| Platform | Reported crawl-to-referral ratio | Measurement period and caveat |
|---|---:|---|
| Anthropic | 70,900:1 | 19–26 June 2025; missing app referrers may create upward bias.[18] |
| Mistral | 0.1:1 | 19–26 June 2025; Cloudflare said Mistral sent ten times as many referrals as crawl requests.[18] |

These figures are historical examples, not November 2026 readings. Cloudflare says native-app referrals without a `Referer` header are not counted, so the ratio may overstate crawling relative to referrals by an unknown amount.[18] A separate Cloudflare analysis reported 38,000 pages crawled per referred page visit for Anthropic in July 2025 and a Perplexity increase from 54 bots per human in January to 195 in July.[20] Those figures use different dates and units and should not be combined into one time series.

## How to configure Cloudflare AI bot controls

The precise interface can change, but the following process reflects Cloudflare’s documented control model.

1. **Proxy the domain through Cloudflare.** AI Crawl Control requires traffic to pass through Cloudflare so requests can be inspected and controlled.[12]
2. **Open the AI crawler or Security settings.** Review the current actions for Search, Agents, and Training rather than assuming a default.[4][2]
3. **Choose an action for each use category.** Use Allow, Block, or ad-page blocking where available; Training also has a Disallow option.[2]
4. **Review individual crawler policies.** Check which crawlers are allowed, blocked, or producing unsuccessful requests.[2]
5. **Decide whether Content Signals are needed.** Use them to express search, AI-input, and training preferences in robots.txt.[3][16]
6. **Use blocking for enforcement.** Do not rely on robots.txt or Content Signals when access must be technically denied.[2]
7. **Check for conflicting rules.** WAF, Bot Management, origin errors, and other responses can affect the result shown as unsuccessful.[7]
8. **Test from representative paths.** Check public content, ad-supported pages, restricted content, and the standard free paths used by Pay Per Crawl.[14]
9. **Export or query analytics.** Use the dashboard or GraphQL API to review crawler, status, path, and referral dimensions.[9]


Which Cloudflare setting should be checked first?

Open the zone’s current AI crawler access policy in Security settings. The public documentation gives conflicting descriptions of defaults for new domains, so the live zone configuration is more reliable than a general statement about Cloudflare’s default behavior.[4]



## Which option should you use?

Use **AI Crawl Control blocking** when the primary requirement is to deny identified AI crawler requests that reach the site through Cloudflare.[2]

Use **Content Signals** when the goal is to communicate differentiated permissions—for example, allowing search indexing while declining AI input or model training.[3][14]

Use **Pay Per Crawl** only when the site is eligible for the beta and the commercial model has been evaluated. It may provide a structured payment flow, but Cloudflare has not published a universal rate or general-availability date in the cited documentation.[5]

Use **analytics** to understand request patterns and possible referral activity, not to claim that every crawl generated value. Referrer headers can be absent, especially when activity comes through native applications.[18]

For most sites, the practical policy is layered:

- Set explicit Search, Agents, and Training actions.
- Use Content Signals to state preferred uses.
- Apply Cloudflare blocking where requests must be denied.
- Review crawler analytics and unsuccessful requests.
- Treat Pay Per Crawl as an optional beta experiment.

## FAQ
### Does Cloudflare AI Crawl Control block AI bots by default?
Not consistently according to Cloudflare’s public documentation. A policy page updated on 2 November 2026 says new domains allow Search, Agents, and Training by default, while Cloudflare’s September 2026 press release describes different defaults for ad-supported sites.[7] Check the settings for the specific zone.

### How do I block AI bots with Cloudflare?
Route the domain through Cloudflare, open the AI crawler or Security settings, and apply block actions to the relevant crawler categories or individual crawlers.[4] Test the result on representative URLs because other WAF, Bot Management, origin, or response rules can affect request outcomes.[7]

### Is Cloudflare Pay Per Crawl available to everyone?
No. Cloudflare’s overview describes Pay Per Crawl as a private beta, and its feature documentation calls it a closed beta.[2][5] The cited sources do not establish a general-availability date or a universal price.

### How much does Cloudflare Pay Per Crawl cost?
Cloudflare does not publish a universal price in the cited documentation. Site owners can configure a zone price, and the documentation shows `USD 3.14` as an example rather than a standard rate.[14] Advanced settings can also support dynamic pricing through an origin `crawler-price` response header.[15][14]

### Do Cloudflare Content Signals stop AI crawlers?
No. Content Signals express whether content is intended for search, AI input, or training, but they do not technically prevent access.[3] Cloudflare recommends an AI Crawl Control block when the requirement is enforcement.

### Can Cloudflare measure whether AI crawlers send valuable traffic?
Cloudflare can measure crawler requests and some referrals, but its crawl-to-referral ratio is not a complete value or conversion metric.[18] Referrals without a `Referer` header, including some native-app activity, are not counted, and the published numerical examples are from 2025.[18]

## Sources
[1] AI Crawl Control — https://developers.cloudflare.com/ai-crawl-control/
[2] Manage AI crawlers — https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers
[3] Content Signals — https://contentsignals.org/
[4] AI Crawler and Agent access policies — https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots
[5] What is Pay Per Crawl? — http://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl
[6] Attribution Business Insights — https://developers.cloudflare.com/bots/attribution-business-insights
[7] Have it both ways: stay discoverable in search while disallowing AI training — https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/
[8] Get started — https://developers.cloudflare.com/ai-crawl-control/get-started
[9] GraphQL API — https://developers.cloudflare.com/ai-crawl-control/reference/graphql-api
[10] New options to manage AI traffic — https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options
[11] robots.txt setting — https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt
[12] Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study — https://arxiv.org/pdf/2505.21733
[13] Pay Per Crawl FAQ — https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq
[14] Advanced configuration — https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/advanced-configuration
[15] Pay Per Crawl advanced configuration · Changelog — https://developers.cloudflare.com/changelog/post/2026-06-16-pay-per-crawl-advanced-configuration/
[16] Giving users choice with Cloudflare’s new Content Signals Policy — https://blog.cloudflare.com/content-signals-policy
[17] Your site, your rules: new AI traffic options for all customers — https://blog.cloudflare.com/content-independence-day-ai-options
[18] The crawl before the fall… of referrals: understanding AI’s impact on content providers — https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar
[19] Unmasking the crawls with Attribution Business Insights — https://blog.cloudflare.com/attribution-business-insights
[20] The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals — https://blog.cloudflare.com/crawlers-click-ai-bots-training
