robots.txt for AI Crawlers in 2026: Training Bots, Search Bots and User Agents Explained

🛡️ Crawlers · Updated October 2026 · 5 min read · by Jacob Campbell

"Should I block AI bots?" is the wrong question, because there is no single kind of AI bot. The major AI companies now run separate crawlers for three different jobs: collecting training data, building a search index, and fetching a page because a user asked. Blocking all three keeps you out of training sets, and also out of the answers people get when they ask an assistant about you. This guide lists the crawlers that matter, explains the three jobs, gives copy-ready rules for the common choices, and covers the part most guides skip: verifying that a bot is who it says it is.

Key facts

• 3 jobs — Training, search indexing and user-triggered fetching
• RFC 9309 — The Robots Exclusion Protocol standard, published 2022
• Advisory — robots.txt is a request, not an access control
• Verify — User agents can be faked; IPs and DNS cannot easily be

The three jobs

• Training crawlers collect public pages to train future models. Blocking them keeps future content out of training data. It does not remove anything already collected.
• Search crawlers build the index an assistant searches when it answers with sources. Blocking them makes you much less likely to be cited or linked.
• User-triggered fetchers load a specific page because a person asked the assistant to (for example by pasting your URL). Providers describe these as acting on behalf of a user, and some state that robots.txt rules may not apply to them in the same way, since a person initiated the request.

The crawlers that matter

From the providers’ own documentation (names change; check the linked pages before relying on them):

• OpenAI (docs): GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (user-triggered).
• Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-triggered).
• Perplexity: PerplexityBot (search index), Perplexity-User (user-triggered).
• Google: Google-Extended is a robots.txt token, not a separate crawler. It controls whether content crawled by Googlebot is used for Gemini models and grounding. It does not affect Google Search, and AI Overviews are part of Search, governed by Googlebot.
• Apple: Applebot-Extended, a token controlling use of Applebot-crawled content for Apple’s generative models.
• Others: CCBot (Common Crawl, a dataset widely used in training), Meta-ExternalAgent, Bytespider (ByteDance), Amazonbot.

Policy A: visible in AI answers, out of training

The most common choice for businesses:

# Training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

# Search and user-triggered: yes
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

User-agent: *
Disallow: /wp-admin/

Sitemap: https://example.com/sitemap.xml

Grouping several User-agent lines above one set of rules is valid under RFC 9309.

Policy B: fully open

If you want the widest reach (most marketing sites, documentation, open-source projects), allow everything and say so in your llms.txt. This is what we do on this site, because being included in training data is, for a web agency, a form of distribution.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Policy C: closed

For paywalled or licensed content, block training and search crawlers and consider licensing signals such as RSL (Really Simple Licensing), an emerging standard for stating machine-readable licence terms; we publish an RSL file on this site as an experiment. Understand the trade-off: you will rarely be cited in AI answers.

The CDN layer can override you

Your robots.txt is not the only gate. Cloudflare and other CDNs offer one-click AI-bot blocking, and Cloudflare can manage robots.txt for you, adding its own rules. In our own agent-readiness experiment we found the CDN’s managed robots.txt and the site’s file could disagree. Always check the robots.txt actually served at your public URL, and check the CDN’s bot settings.

Verifying that a bot is real

robots.txt is advisory, and anyone can send User-Agent: GPTBot. If you treat bots differently (allowing more, rate-limiting less), verify them:

• Published IP ranges. OpenAI and others publish JSON lists of the IP ranges their crawlers use. Check the request IP against the list.
• Reverse and forward DNS. Googlebot and Bingbot can be verified by a reverse DNS lookup to the provider’s domain followed by a forward lookup back to the same IP.
• Signed requests. Web Bot Auth, based on HTTP Message Signatures (RFC 9421), lets bots sign requests cryptographically; Cloudflare’s verified-bot programme supports it.

Unverified requests claiming to be a known AI bot should get the rules for an unknown visitor. The AI Indexing Pack includes bot access control that verifies crawler identity this way, so a spoofed user agent does not get a verified bot’s treatment.

Testing your rules

1. Fetch your live robots.txt from outside your network.
2. Use Google Search Console’s robots.txt report for Googlebot.
3. Check server logs for each AI user agent and the status codes they received.
4. Run WebAgentScan, which reports which AI agents your robots.txt allows.

Related reading

• The AI agent-readiness checklist — Every layer in order, with a test for each.
• Agent discovery files explained — agents.txt, agent cards, server cards and the .well-known tree.
• How to get cited by AI assistants — What actually makes ChatGPT, Claude and Perplexity name you.
• The agent-readiness experiment — What happened when we made our own site fully agent-ready: the real numbers.
• Make WordPress readable by AI — llms.txt, Markdown, MCP and crawler access on WordPress.
• AI Indexing Pack — The file-drop kit: MCP server, Markdown pages, llms.txt, discovery files and bot access control for any PHP site.
• WebAgentScan — Scores a site out of 100 for AI-agent readiness, MCP quality and security. Paid plans from $9 a month.

Frequently asked questions

Should I block GPTBot?

Blocking GPTBot stops OpenAI using your future content for training. It does not stop ChatGPT search (OAI-SearchBot) or user-triggered fetches (ChatGPT-User), which have separate user agents.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls use of content for Gemini models and grounding. AI Overviews are part of Google Search and are governed by Googlebot.

What is the difference between ClaudeBot and Claude-User?

ClaudeBot collects public content for training. Claude-SearchBot indexes for search. Claude-User fetches a page when a user asks Claude to.

Is robots.txt legally binding?

robots.txt is an advisory standard (RFC 9309), not an access control. Reputable crawlers follow it, but it cannot technically stop a request.

How do I verify that a request really comes from an AI crawler?

Check the IP against the provider's published IP ranges, use reverse and forward DNS where the provider supports it, or rely on signed requests such as Web Bot Auth.

Can my CDN override my robots.txt?

Yes. CDN bot-blocking features and managed robots.txt can block or add rules. Always check the robots.txt served at your public URL.