Cloudflare and AI Bots: Blocking, AI Crawl Control and Pay Per Crawl Explained

🛡️ AI Bots · Updated October 2026 · 12 min read · by Jacob Campbell

If your site sits behind Cloudflare, Cloudflare may already be deciding which AI bots can read it: since July 1, 2025 new domains are asked whether to allow AI crawlers at sign-up, and since September 15, 2026 new domains get defaults that block AI Training and Agent bots on pages that show ads. Every plan, including Free, can now set separate policies for Search, Agent and Training bots, block or allow individual crawlers in AI Crawl Control, and publish a managed robots.txt with Content Signals.

This guide explains each feature, the one setting that can quietly block Googlebot and Bingbot, how Pay Per Crawl and HTTP 402 work, what “verified bot” and Web Bot Auth mean, and a step-by-step setup for a small business on the Free plan that wants to be found by search and AI answer engines.

Key facts

• Jul 1, 2025 — Cloudflare starts asking every new domain whether to allow AI crawlers
• 3 behaviours — Search, Agent and Training: separate AI bot policies on every plan since July 2026
• Sep 15, 2026 — Mixed Search + Training crawlers become subject to Training blocks
• HTTP 402 — Payment Required: the response Pay Per Crawl returns with a crawler-price header

What Cloudflare changed, and when

Cloudflare’s AI bot controls have gone from an opt-in dashboard in 2024 to behaviour-based defaults in 2026, and the settings you see depend partly on when your domain joined.

• September 23, 2024: AI Audit became available to every Cloudflare site, showing which AI services crawl you and letting you block them.
• July 1, 2025: Cloudflare announced that “upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers,” and introduced Pay Per Crawl as a private beta. Press coverage summarised this as blocking AI crawlers by default.
• August 27, 2025: AI Audit was renamed AI Crawl Control and became generally available.
• September 24, 2025: Cloudflare launched the Content Signals Policy for robots.txt (search, ai-input, ai-train).
• July 1, 2026: all customers, including Free, gained separate policies for Search, Agent and Training bots, each set to block on all pages, block only on pages with ads, or allow. Signed agents were folded into Verified bots the same day.
• September 15, 2026: new domains onboarding to Cloudflare get Training and Agent blocked on pages that display ads, with Search allowed. Mixed-purpose crawlers are now judged by all of their behaviours, and the older single “Block AI bots” toggle is being deprecated.

Sources: Cloudflare’s AI Crawl Control documentation, its Block AI Bots reference and the changelogs linked from them.

The Googlebot trap: blocking Training can block search

Since September 15, 2026, any Cloudflare setting that blocks Training also blocks crawlers that Cloudflare classifies as both Search and Training, and Cloudflare names Googlebot, Applebot and BingBot as examples.

Cloudflare’s documentation says mixed-purpose crawlers “will also be blocked by all configurations to block AI training, including the legacy ‘Block AI bots’ option,” and its July 2026 blog post explains that defaults are “enforced by the most restrictive applicable rule.” Before that date, the legacy toggle excluded mixed-purpose bots.

For a business that depends on Google and Bing for customers, that means:

• Do not set the Training policy to “Block (on all pages)” unless you have decided that keeping your content out of training matters more than search traffic.
• “Block on pages with ads” limits the effect to pages Cloudflare detects as showing ads, which for most service businesses is none, but check what your new-domain default is.
• To refuse training without losing search, block training-only crawlers individually in AI Crawl Control (for example GPTBot or CCBot) and use the robots.txt product tokens Google-Extended and Applebot-Extended, which control use for AI models without affecting the search crawlers. Our AI crawler user-agent list says which bot does what.
• Verify, do not assume. Use the URL Inspection live test in Google Search Console and the equivalent in Bing Webmaster Tools to confirm the crawler gets a 200, not a 403.

AI Crawl Control: see and set access per crawler

AI Crawl Control is the Cloudflare dashboard section (formerly AI Audit) that shows which AI crawlers request your pages and lets you allow or block each one; it is available on all plans.

It has four main tabs:

• Overview: a snapshot of AI crawler activity, filterable by date, crawler, operator, hostname and path.
• Crawlers: one row per crawler with its operator, category, allowed and unsuccessful requests, robots.txt violations, and an Action column set to Allow or Block.
• Metrics: breakdowns by crawler, operator, status code, hostname and path, plus referrals from AI platforms on paid plans. On the Free plan this tab covers only the last 24 hours.
• Directives (renamed from Robots.txt in April 2026): whether your robots.txt is reachable, whether it contains Content Signals, which crawlers requested disallowed paths, and a link to an agent-readiness check.

Three details matter in practice. First, on the Free plan AI Crawl Control identifies crawlers by their user-agent string, so it catches well-known, self-identifying bots; paid plans add Bot Management detection. Second, blocking crawlers creates or updates one WAF custom rule on your zone, which counts toward your plan’s custom-rule allowance, and earlier WAF rules can still block crawlers you set to Allow. Third, customising the block response (a 402 Payment Required or 403 Forbidden with your own message) is a paid-plan feature.

Pro plans and above can also turn on Redirects for AI Training, which sends verified training crawlers to the URL in your <link rel="canonical"> while humans and search crawlers see the page normally; see canonical tags explained.

Managed robots.txt and the Content Signals Policy

Cloudflare’s managed robots.txt setting adds AI crawler rules and Content Signals to the robots.txt it serves for your domain, prepending them to your own file if you have one.

Turn it on under Security → Settings, filter by Bot traffic, and enable Set your preference to block training in robots.txt. Cloudflare’s documentation shows the managed block it adds:

# BEGIN Cloudflare Managed content

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# END Cloudflare Managed Content

Above that block Cloudflare also adds the Content Signals Policy text, which defines search, ai-input (using content for real-time AI answers and grounding) and ai-train, and states that restrictions are reservations of rights under Article 4 of the EU copyright directive. The use=reference field is a newer, experimental extension (index, excerpt and link back). Free-plan domains without their own robots.txt and without the managed setting are served the policy text with no preferences filled in.

Two cautions. robots.txt is a request, not enforcement; Cloudflare itself says some operators ignore it, which is why AI Crawl Control blocks exist. And Google Search Console may report “Syntax not understood” for the Content-signal line; Cloudflare says it has seen no effect on crawling. Read Content Signals and RSL for what the signals mean and robots.txt for AI crawlers for writing your own.

Pay Per Crawl and HTTP 402

Pay Per Crawl lets a site owner charge AI crawlers a set price per successful request, with Cloudflare acting as Merchant of Record; it is still a closed beta that you apply to join.

The mechanics, from Cloudflare’s documentation:

• A crawler requesting a charged page without payment intent gets HTTP 402 Payment Required and a crawler-price header, for example crawler-price: USD 0.01.
• The crawler retries with crawler-exact-price (the quoted price) or crawler-max-price (a ceiling), receives 200, and a crawler-charged header confirms the amount.
• Crawlers must sign their requests with Web Bot Auth, and the payment header must be one of the signed components.
• The minimum price is $0.001 per crawl. Site owners can make specific paths free with Configuration Rules or price dynamically from the origin; /robots.txt, /sitemap.xml and security.txt paths are always free. Payouts go through Stripe.
• WAF and Bot Management blocks run first, so a blocked crawler is never offered a price.

For a local business, Pay Per Crawl is mostly a signal of where the market is heading rather than a revenue line: you need beta access, crawlers that agree to pay, and content valuable enough to charge for. Most small sites gain more from being read and cited than from a fraction of a cent per page.

Verified bots, Web Bot Auth and signed agents

A Cloudflare Verified bot is a bot or agent Cloudflare has confirmed identifies itself honestly and behaves well, and verification is what lets Cloudflare tell the real Googlebot or GPTBot from a scraper copying its user-agent.

Cloudflare’s two bars are honest self-identification (a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS) and non-abusive behaviour (obeying robots.txt, reasonable request rates, no evasion). Verified bots are listed in Cloudflare Radar’s bots and agents directory.

Web Bot Auth is the cryptographic method. It builds on two IETF drafts: the bot publishes its Ed25519 public keys at /.well-known/http-message-signatures-directory, and signs each request with HTTP Message Signatures (RFC 9421) using Signature and Signature-Input headers. A signature cannot be faked by changing a header, which is why Pay Per Crawl requires it.

Signed agents were a separate category for AI agents acting for a user. Since July 1, 2026 they are Verified bots too, labelled Direct (one narrow operator) or Intermediary (an agentic service that many end users drive). As a site owner you do not need to implement Web Bot Auth; it matters because Verified status is what the Search, Agent and Training presets act on most reliably.

Step by step: a sensible setup on the Free plan

For a small business that wants customers to find it through Google, Bing and AI answer engines, the goal is to allow Search and Agent traffic, decide deliberately about Training, and confirm nothing important is blocked.

1. Check your AI bot policies. In the dashboard go to Security → Settings and open Configure AI bot policies. Set Search to Allow and Agent to Allow, so assistants fetching your pages for a user’s live question can read them. If your domain joined after September 15, 2026, check whether Training and Agent were defaulted to “Block on pages with ads”.
2. Decide on Training without the trap. Leave the Training preset on Allow if you rely on Google and Bing, and if you do not want your content used for training, block the training-only crawlers you object to in AI Crawl Control → Crawlers instead.
3. Turn off the legacy toggle. If Block AI bots is still on from an earlier setup, replace it with the presets; it is being deprecated and now also catches mixed-purpose crawlers.
4. Publish your preferences. Enable managed robots.txt, or write your own file with a Content-signal line and Google-Extended / Applebot-Extended rules. Fetch it to see exactly what Cloudflare serves: curl -s https://example.com/robots.txt.
5. Be careful with Bot Fight Mode. It challenges traffic matching bot patterns, and Cloudflare notes it cannot be skipped with WAF custom rules or Page Rules. If assistants report they cannot read your pages, test with it off.
6. Check what is blocked. Review the Action column in AI Crawl Control, the status-code chart in Metrics (anything other than 200 for a crawler you want), robots.txt violations in Directives, and Security → Events for the AI Crawl Control WAF rule and any other rule that fired.
7. Confirm from the outside. Run Search Console’s live URL test and Bing’s URL inspection, then look for AI bots in your own server logs. Repeat after any Cloudflare change.

Limits: what Cloudflare controls and what it cannot

Cloudflare can only act on requests that pass through its proxy and bots it can recognise, and its categories are its own judgement, not a standard.

• Grey-clouded records bypass it. Only proxied (orange-cloud) hostnames get these controls; a DNS-only subdomain or a direct origin IP does not.
• Free-plan detection is user-agent based. Crawlers that disguise themselves as browsers are not caught by AI Crawl Control on Free.
• Classification can change. Which bots count as Search, Agent or Training is decided by Cloudflare, and a reclassification can change who your settings block without you touching anything. Re-check after Cloudflare announcements.
• robots.txt and Content Signals are preferences. They express terms; they do not stop a crawler that ignores them.
• Blocking is not visibility. Allowing every AI bot does not make assistants cite you; that depends on content, structured data and being findable. See how to get cited by AI assistants.

Product facts: the AI Indexing Pack

If your site is not behind Cloudflare, the pack includes its own bot access control that verifies crawler identity. Stated plainly, so it can be checked against the product page:

• What it is: the AI Indexing Pack is a set of files you upload to your own website. It is sold by Built2WinWeb as a one-time purchase, with no subscription.
• What it ships: (1) a working MCP server using the Streamable HTTP transport, with typed output schemas and tool annotations; (2) Markdown content negotiation, so agents can request a clean text version of each page; (3) llms.txt, agents.txt and agent cards; (4) the full /.well-known/ discovery tree; (5) OAuth protected-resource metadata; (6) bot access control that verifies crawler identity; (7) a README with a verification checklist.
• Requirements: PHP 7.4 or newer, Apache or LiteSpeed with .htaccess support, FTP access, and a domain you control. Installation takes about 30 minutes.
• What it is not: it is not a WordPress plugin (that is the separate WordPress edition, which needs WordPress 6.0+ and PHP 8.0+), it does not write content for you, it does not guarantee rankings or citations, and it is not a managed service.
• Free routes in: sites built on AIWebBuilding get the core built in, and a 50-day streak on FreeAIBoost unlocks the pack for one domain. Current prices are on the product pages.

Related reading

• AI crawler user-agent list (2026) — Every AI bot, who runs it and whether it trains, searches or fetches for users.
• robots.txt for AI crawlers — Which bots to allow, which to block, with copy-ready rules.
• Content Signals and RSL — Telling AI systems how they may use your content.
• Find AI crawlers in your server logs — Confirm which bots actually reach your site and what they get back.
• Bing Webmaster Tools for AI search — Check Bingbot access and your visibility in Microsoft’s AI answers.
• How AI assistants read a web page — Why bot protection is one of the most common reasons AI fetches fail.
• AI Indexing Pack — The file-drop kit: MCP server, Markdown pages, llms.txt, discovery files and bot access control for any PHP site.

Frequently asked questions

Does Cloudflare block AI crawlers by default?

For new domains, partly. Since July 1, 2025 new domains are asked at sign-up whether to allow AI crawlers, and since September 15, 2026 new domains default to blocking Training and Agent bots on pages that display ads while allowing Search.

Can blocking AI training on Cloudflare block Googlebot?

Yes. Since September 15, 2026 Cloudflare applies Training blocks to mixed-purpose crawlers that also do Search, and it names Googlebot, Applebot and BingBot. Block training-only crawlers individually instead if you need search traffic.

What is AI Crawl Control?

AI Crawl Control, formerly AI Audit, is the Cloudflare dashboard section that shows which AI crawlers request your pages and lets you allow or block each one. It is available on all plans; Free-plan detection is based on user-agent strings.

How does Pay Per Crawl work?

A crawler requesting a charged page gets HTTP 402 Payment Required with a crawler-price header, then retries with a signed crawler-exact-price or crawler-max-price header to receive the page and be billed. It is a closed beta with a $0.001 minimum price.

How do I see what Cloudflare is blocking?

Check the Action column and Metrics status codes in AI Crawl Control, the Directives tab for robots.txt issues, and Security Events for the WAF rule that fired. Confirm with the live URL test in Google Search Console.