Free tool · No sign-up

AI Crawler Checker

Find out which AI bots your robots.txt lets in, and which line decides it. 27 crawlers from OpenAI, Anthropic, Google, Perplexity, Apple, Meta and more, tested the way they actually read the file.

Check a robots.txt file

We fetch https://your-domain/robots.txt once and test it. Public sites only.

Options: test a specific page or add your own bot

A path like /blog/post/ or a full URL.

Comma-separated, up to five.

Try:

What this AI crawler checker tests

Most robots.txt testers check one bot at a time, and most "AI bot" lists stop at GPTBot. The decision that matters now is different: AI companies split their traffic into separate crawlers for training, for search and for fetching a page a user asked about, and each one reads your robots.txt on its own. This tool runs all of them against the same file and the same page, so you can see your policy as a whole.

How the file is read

Exactly as RFC 9309 and Google specify. A crawler obeys only the most specific User-agent group that names it, falling back to User-agent: *; groups that name the same bot are merged; the longest matching Allow or Disallow wins, with Allow winning a tie; * and $ work as wildcards. A missing robots.txt (404) means everything is allowed; a server error (5xx) makes compliant crawlers treat the whole site as blocked, which the tool flags as an error.

What gets flagged

How to fix what you find

The table's Decided by column gives the line number behind every result: edit that line, paste the new file into the Paste robots.txt tab to confirm the change does what you expect, then deploy and re-check the live domain. If you are choosing a policy from scratch, start from one of these.

In AI search, out of training 
# Stay in AI search, opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

The most common choice for businesses: be found and cited, without feeding training sets.

Open to all 
# Welcome every crawler, AI included
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Maximum visibility. Right for most small businesses that want models to know who they are.

Search engines only 
# Keep Google, Bing and Apple search; block AI crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Keeps Google, Bing and Apple. Some user-triggered fetchers may still visit; enforcing that needs a firewall rule.

For every bot's operator, purpose and documentation, see the AI crawler user-agent list.

Letting AI in is half of it. Being readable is the other half.

Once the right crawlers can reach you, what they find decides whether they can use it: clean Markdown, an llms.txt, structured data, an MCP server.

Score the whole site

from $9/month

WebAgentScan audits every agent surface, verifies bot access, checks MCP security and tracks the score over time.

See WebAgentScan

Install the fix

$39 once · WordPress $49 once

The AI Indexing Pack includes bot access control that verifies a crawler is who it claims, plus llms.txt, Markdown negotiation and an MCP server.

See the pack

Check your llms.txt

Free

The companion tool: validates your llms.txt against the format and checks how it is served.

llms.txt Validator

Further reading

AI crawler FAQ

Should I block GPTBot and ClaudeBot?

Only if you do not want your content used to train models. Blocking them does not remove you from ChatGPT or Claude answers: those come from separate search crawlers (OAI-SearchBot, Claude-SearchBot) and user-triggered fetchers. Many businesses want to be known to models and leave training open; others block it. Either is fine, as long as it is a decision rather than an accident.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended is a token that controls use of your content for Gemini training and grounding. AI Overviews and AI Mode are built from the normal Search index that Googlebot crawls. Google offers a separate Search Console setting for those features.

Why is a crawler "allowed" when I never mentioned it?

Because robots.txt is allow-by-default. A crawler with no group of its own follows the User-agent: * group, and if no rule there matches the path, it may fetch it. The "Decided by" column shows which group and line applied.

How does the checker decide between Allow and Disallow?

The same way RFC 9309 and Google do: the crawler uses only the most specific group naming it (or *), then the longest matching rule wins. If an Allow and a Disallow match with the same length, Allow wins. * matches any characters and $ anchors the end of the URL.

Do AI crawlers actually obey robots.txt?

The training and search crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Mistral, DuckDuckGo and Common Crawl say they do. Several user-triggered fetchers say robots.txt may not apply because a person asked for the page. To enforce a block on bots that ignore it, you need a firewall rule, not robots.txt.

Is anything I check stored?

No. Pasted robots.txt is checked entirely in your browser. For a domain, our server fetches that one public file, returns it to your browser and keeps nothing.