AI Crawler User-Agent List (2026): Every Bot, Who Runs It and What It Does

🤖 AI Crawlers · Updated October 2026 · 11 min read · by Jacob Campbell

AI companies send three different kinds of bot to your website: training crawlers that collect content for future models, search crawlers that build an index the assistant answers from, and user-triggered fetchers that load one page because a person just asked about it. Each kind has its own user-agent name, and blocking one does not block the others. This page lists the AI-related bots that their operators document publicly as of October 2026, with what each one is for, whether the operator says it follows robots.txt, and how to check that a request claiming to be that bot is genuine. Every entry was checked against the operator’s own documentation; bots we could not verify that way are left out and named at the end.

Key facts

• 3 kinds — Training crawlers, search crawlers and user-triggered fetchers
• ~24 hours — Time OpenAI and Perplexity say a robots.txt change can take to apply
• 0 requests — Sent under the Google-Extended and Applebot-Extended names: they are robots.txt tokens, not crawlers
• 72 hours — DuckDuckGo’s stated delay before a DuckAssistBot opt-out takes effect

Training, search and user-triggered: the three kinds of AI bot

The single most useful thing to know about AI crawlers is that most large operators now split their traffic into separate user agents by purpose, so you can make a different decision for each purpose.

• Training crawlers collect pages that may be used to train future models. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), MistralAI-Training, meta-externalagent. Blocking them does not remove you from any assistant’s live answers.
• Search crawlers build the index an assistant searches when it answers a question and links to sources. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, MistralAI-Index, Googlebot. Blocking these is what removes you from AI search answers.
• User-triggered fetchers load a single URL because a person asked the assistant to read it, or because the assistant decided to open a page while answering. Examples: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, meta-externalfetcher, Google-Agent. Several operators say these may not follow robots.txt, on the reasoning that a human initiated the request.

There is a fourth category that is not a bot at all: robots.txt product tokens. Google-Extended and Applebot-Extended never appear in your logs. They are names you write in robots.txt to tell Google or Apple how content their normal crawlers already fetched may be used for AI. For the policy side of these choices, see robots.txt for AI crawlers.

The list at a glance

This table lists the user-agent token to use in robots.txt, the operator, what the bot is for, and what the operator’s documentation says about robots.txt. Details, full user-agent strings and verification methods follow in the next sections.

OpenAI and Anthropic

OpenAI documents four user agents and states that “each setting is independent of the others”, so you can allow search while refusing training. Source: OpenAI crawler overview.

• GPTBot crawls content that may be used in training OpenAI’s foundation models. Disallowing it signals that your content should not be used for training. UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. IP ranges: https://openai.com/gptbot.json.
• OAI-SearchBot surfaces websites in ChatGPT’s search features. OpenAI says sites that block it will not appear in ChatGPT search answers, and that a robots.txt change can take about 24 hours to take effect. IP ranges: https://openai.com/searchbot.json.
• ChatGPT-User handles user-initiated actions in ChatGPT and custom GPTs. OpenAI says it is not used to crawl automatically, is not used to decide what appears in Search, and that “robots.txt rules may not apply”. IP ranges: https://openai.com/chatgpt-user.json.
• OAI-AdsBot visits only ad landing pages submitted to OpenAI, to check safety and relevance; robots.txt does not apply. IP ranges: https://openai.com/adsbot.json.

Anthropic documents three bots in its help centre article on how it crawls the web, and says all of them respect standard robots.txt directives and the non-standard Crawl-delay extension.

• ClaudeBot collects web content for model training. Blocking it signals that future content should be excluded from training datasets.
• Claude-SearchBot crawls to improve search result quality. Blocking it prevents your content being indexed for Claude’s search.
• Claude-User fetches pages when a Claude user asks a question. Blocking it prevents Claude retrieving your pages in response to user queries.

Anthropic states that it does not currently publish IP ranges, because it uses service-provider public IPs, so IP blocking may not reliably work for its bots. Use the user-agent name in robots.txt.

Perplexity, Mistral and DuckDuckGo

Perplexity runs one search crawler and one user-triggered fetcher, documented on its crawlers page.

• PerplexityBot surfaces and links websites in Perplexity search results and, per Perplexity, “is not used to crawl content for AI foundation models”. It respects robots.txt; changes may take up to 24 hours to apply. UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). IPs: https://www.perplexity.com/perplexitybot.json.
• Perplexity-User visits pages to answer a user’s question and “generally ignores robots.txt rules” because a user triggered the fetch. IPs: https://www.perplexity.com/perplexity-user.json.

Mistral separates all three purposes, per its robots page:

• MistralAI-User visits a page when a user asks Mistral’s assistant a question and links the source; Mistral says it is not used for training and that robots.txt rules for it govern which sites these user requests can reach. IPs: https://mistral.ai/mistralai-user-ips.json.
• MistralAI-Index crawls for indexing only, to support search, and is “not used for generative AI training of any kind”. IPs: https://mistral.ai/mistralai-index-ips.json.
• MistralAI-Training builds datasets for training Mistral models and can be disallowed in robots.txt.

DuckDuckGo’s DuckAssistBot (UA DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)) crawls pages in real time for AI-assisted answers that cite their sources. DuckDuckGo states the data is not used to train AI models, that a robots.txt opt-out takes effect after 72 hours, and that opting out does not affect organic rankings. IPs: https://duckduckgo.com/duckassistbot.json. Source: DuckDuckGo help.

Google, Apple and Microsoft: tokens versus crawlers

Google, Apple and Microsoft mostly reuse their search crawlers for AI features and give you separate controls instead of separate bots.

Google. AI Overviews and AI Mode are built from the normal Google Search index, which Googlebot crawls; Google says a page must be indexed and eligible for a snippet to appear as a supporting link, so blocking Googlebot removes you from Search entirely. To leave AI Overviews and AI Mode without leaving Search, Google now offers a Search Console setting instead of a bot name: Settings > Search generative AI, rolled out to all sites worldwide as of August 31, 2026, which Google says is not used as a ranking signal for the rest of Search (Search generative AI control). Google-Extended is, in Google’s words, “a standalone product token” that controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It has no separate user-agent string and “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. Google-CloudVertexBot crawls only when a site owner asks it to, to build Vertex AI Agents. Google-Agent is a user-triggered fetcher “used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request”; like Google’s other user-triggered fetchers, it generally ignores robots.txt. Sources: Google’s common crawlers and user-triggered fetchers.

Apple. Applebot powers search in Spotlight, Siri and Safari, and Apple says the data may also be used to train its foundation models. If your robots.txt has no Applebot rules but does have Googlebot rules, Applebot follows the Googlebot rules. Applebot-Extended does not crawl; disallowing it tells Apple not to use your content to train generative models, while your pages can still appear in Apple’s search results. Source: About Applebot.

Microsoft. Microsoft does not document a separate AI crawler; bingbot crawls for Bing. Instead, Microsoft announced in September 2023 that the NOCACHE and NOARCHIVE robots meta values control use of content in what was then Bing Chat (now Copilot) and in training its generative models, while pages stay in Bing search results. Bing Webmaster Tools now reports how often your pages are cited in Copilot and Bing’s AI summaries (see measuring AI search traffic).

Meta, Amazon and Common Crawl

Meta documents several crawlers on its web crawlers page, and recommends allow-listing by IP address rather than by user-agent string:

• meta-externalagent crawls for training AI models or improving products by indexing content directly. Respects robots.txt.
• meta-webindexer crawls to improve Meta AI search result quality. Respects robots.txt.
• meta-externalfetcher fetches individual links at a user’s request; Meta says it may bypass robots.txt for that reason.
• facebookexternalhit fetches pages shared in Meta’s apps to build link previews, and may bypass robots.txt during security or integrity checks.

Amazon documents three agents on its Amazonbot page, each with its own IP list:

• Amazonbot improves Amazon products and services and “may be used to train Amazon AI models”. It honours robots.txt. IPs: https://developer.amazon.com/amazonbot/ip-addresses/.
• Amzn-SearchBot makes content eligible for Amazon search experiences such as Alexa and does not crawl for generative AI training. IPs: https://developer.amazon.com/amazonbot/searchbot-ip-addresses/.
• Amzn-User fetches live information for user actions such as Alexa questions and “may not follow all robots.txt directives”. IPs: https://developer.amazon.com/amazonbot/live-ip-addresses/.

CCBot (UA CCBot/2.0 (https://commoncrawl.org/faq/)) is the crawler of Common Crawl, a non-profit that publishes an open repository of web crawl data that anyone can download, including AI developers. Blocking CCBot keeps future crawls of your site out of that dataset; it does not remove copies already published. It respects robots.txt and publishes IP ranges at https://index.commoncrawl.org/ccbot.json, with reverse DNS hostnames under crawl.commoncrawl.org. Source: Common Crawl.

Not in this list: ByteDance’s Bytespider is widely reported in server logs, but we found no current documentation from ByteDance describing it, so we cannot state its purpose or robots.txt behaviour. Treat any undocumented bot by what you can verify: its IP owner and its behaviour in your logs (how to find AI crawlers in your server logs).

How to verify a bot is genuine

A user-agent string is just text that anyone can send, so a request claiming to be GPTBot or Googlebot proves nothing until you match its IP address against what the operator publishes. Operators offer one or both of two methods.

1. Published IP ranges. OpenAI, Perplexity, Mistral, DuckDuckGo, Apple (https://search.developer.apple.com/applebot.json), Amazon, Common Crawl and Google (common-crawlers.json and user-triggered-agents.json, among others) publish lists. This Python 3 script, using only the standard library, checks an IP against any of the JSON lists without depending on a particular file layout:

import ipaddress, json, socket, sys, urllib.request

def networks(obj):
    """Yield every CIDR range found anywhere in a published IP-range JSON file."""
    if isinstance(obj, dict):
        for value in obj.values():
            yield from networks(value)
    elif isinstance(obj, list):
        for value in obj:
            yield from networks(value)
    elif isinstance(obj, str) and "/" in obj:
        try:
            yield ipaddress.ip_network(obj, strict=False)
        except ValueError:
            pass

def ip_in_published_ranges(ip, json_url):
    with urllib.request.urlopen(json_url, timeout=10) as resp:
        data = json.load(resp)
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in networks(data) if net.version == addr.version)

if __name__ == "__main__":
    # python3 verify_bot.py 203.0.113.7 https://openai.com/gptbot.json
    print(ip_in_published_ranges(sys.argv[1], sys.argv[2]))

Cache the downloaded lists (refresh daily) rather than fetching them on every request.

2. Reverse then forward DNS. Google, Apple, Microsoft and Common Crawl support this. Look up the hostname for the IP, check that it ends in the operator’s domain, then look the hostname up again and confirm it returns the same IP:

host 66.249.66.1
# expect a name ending in .googlebot.com (Google), .applebot.apple.com (Apple),
# .search.msn.com (Bing) or .crawl.commoncrawl.org (Common Crawl)
host crawl-66-249-66-1.googlebot.com
# must resolve back to 66.249.66.1

The forward lookup matters: anyone who controls an IP block can set its reverse DNS to say googlebot.com, but they cannot make Google’s DNS point that name back at their IP. Microsoft specifically advises against hardcoding Bingbot IP addresses because they change, so use DNS for Bing.

Anthropic is the exception among the large operators: it publishes no IP ranges, so its bots can only be identified by user agent and controlled through robots.txt.

robots.txt examples

robots.txt groups can list several User-agent lines that share one set of rules (RFC 9309), which keeps an AI policy short. Three common policies:

Stay in AI search, opt out of training.

# Training crawlers and training-control tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

# Everyone else, including OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, Googlebot and bingbot
User-agent: *
Allow: /

Note the trade-offs. Google-Extended also covers grounding in Gemini Apps and Vertex AI, not only training. meta-externalagent is also used for indexing. Amazonbot is left allowed here because Amazon describes it as serving products and services as well as training; add it to the first group if training is your main concern.

Keep a private area away from every AI fetcher that honours robots.txt.

User-agent: *
Disallow: /client-portal/
Disallow: /checkout/

Block one assistant’s search crawler only (for example, if you do not want to appear in ChatGPT search):

User-agent: OAI-SearchBot
Disallow: /

Remember what robots.txt cannot do. It is a request, not access control. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not apply it, and a bot that lies about its name ignores it entirely. If content must not be fetched, put it behind authentication or block it at the server with verified IP rules. And a rule is only as good as the name in it: a typo such as GPT-Bot silently matches nothing. The full policy discussion is in robots.txt for AI crawlers, and Content Signals and RSL covers the newer ways to state usage terms.

Product facts: the AI Indexing Pack

Stated plainly, so it can be checked against the product page:

• What it is: the AI Indexing Pack is a set of files you upload to your own website. It is sold by Built2WinWeb as a one-time purchase, with no subscription.
• What it ships: (1) a working MCP server using the Streamable HTTP transport, with typed output schemas and tool annotations; (2) Markdown content negotiation, so agents can request a clean text version of each page; (3) llms.txt, agents.txt and agent cards; (4) the full /.well-known/ discovery tree; (5) OAuth protected-resource metadata; (6) bot access control that verifies crawler identity; (7) a README with a verification checklist.
• Requirements: PHP 7.4 or newer, Apache or LiteSpeed with .htaccess support, FTP access, and a domain you control. Installation takes about 30 minutes.
• What it is not: it is not a WordPress plugin (that is the separate WordPress edition, which needs WordPress 6.0+ and PHP 8.0+), it does not write content for you, it does not guarantee rankings or citations, and it is not a managed service.
• Free routes in: sites built on AIWebBuilding get the core built in, and a 50-day streak on FreeAIBoost unlocks the pack for one domain. Current prices are on the product pages.

Related reading

• robots.txt for AI crawlers — Which bots to allow, which to block, and the trade-offs of each choice.
• Find AI crawlers in your server logs — Which bots visit, what they fetch and what they get back.
• How AI assistants read a web page — Fetching, JavaScript rendering and content extraction, and how to test your own pages.
• Measure traffic from AI assistants — Referrers, utm_source=chatgpt.com, GA4 channel groups and citation reports.
• Content Signals and RSL — Telling AI systems how they may use your content, beyond allow and disallow.
• AI Indexing Pack — The file-drop kit: MCP server, Markdown pages, llms.txt, discovery files and bot access control for any PHP site.

Frequently asked questions

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

GPTBot collects content that may be used to train OpenAI models, OAI-SearchBot crawls for ChatGPT search results, and ChatGPT-User fetches pages when a user’s request needs them. OpenAI treats each setting independently, so you can block GPTBot and still appear in ChatGPT search.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode draw on the normal Search index. To exclude a site from those features, Google provides the Search generative AI control under Settings in Search Console.

Do AI crawlers obey robots.txt?

The training and search crawlers documented by OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Mistral, DuckDuckGo and Common Crawl say they do. Several user-triggered fetchers, including ChatGPT-User, Perplexity-User, meta-externalfetcher and Google-Agent, say robots.txt may not apply because a person requested the page.

How do I know a request really comes from ClaudeBot or GPTBot?

For GPTBot, check the IP against OpenAI’s published list at openai.com/gptbot.json. Anthropic does not publish IP ranges for ClaudeBot, so it can only be identified by user agent and controlled through robots.txt.

Is Applebot-Extended a separate crawler?

No. Apple says Applebot-Extended does not crawl web pages; it is a robots.txt token that controls whether content Applebot collects may be used to train Apple’s generative AI models.