How to Find AI Crawlers in Your Server Logs: Commands, Patterns and What the Numbers Mean

📊 Analytics · Updated October 2026 · 4 min read · by Jacob Campbell

Analytics tools that run in the browser miss almost all AI traffic, because crawlers and agents do not run your JavaScript. Your web server’s access log is the only place that records every AI crawler visit, what it asked for, and what you sent back. This guide shows how to find AI bots in a standard Apache or Nginx log, how to tell real ones from impostors, and how to read the results as a measure of AI visibility.

Key facts

• 0 — AI crawler visits most JavaScript analytics will show you
• User-Agent — The first filter: who the request says it is
• IP check — The second filter: whether that is true
• Status codes — What you actually served them

Getting the log

On most hosting the raw access log is available from the control panel or over SFTP, usually in the Apache or Nginx “combined” format: IP, time, request line, status, bytes, referrer and user agent. If your site sits behind a CDN such as Cloudflare, the origin log shows CDN IPs; use the CDN’s own logs or the header it adds with the real client IP. Some hosts keep only a few days of logs, so download them regularly.

Finding AI user agents

A single pattern catches the main AI agents (see robots.txt for AI crawlers for what each does):

grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Applebot|Bytespider|Meta-ExternalAgent|Amazonbot|bingbot" access.log > ai.log

Then count visits per bot:

grep -oEi "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|bingbot" ai.log | sort | uniq -c | sort -rn

What they fetch and what they get

Two more one-liners answer the questions that matter:

# Most-requested paths by AI agents
awk '{print $7}' ai.log | sort | uniq -c | sort -rn | head -30

# Status codes returned to AI agents
awk '{print $9}' ai.log | sort | uniq -c | sort -rn

Look for: lots of 403 (your firewall or CDN is blocking them), 404 on llms.txt or discovery files (they are looking and finding nothing), 301 chains (wasted fetches), and 5xx (errors under crawl load).

Separating real bots from fakes

Anyone can send User-Agent: GPTBot. To verify:

• Published IP lists. OpenAI and others publish the IP ranges their bots use. Check each IP against the list.
• Reverse then forward DNS for providers that support it, such as Googlebot and Bingbot: the IP should resolve to the provider’s domain, and that hostname should resolve back to the same IP.
• Behaviour. A “GPTBot” that hammers your login page is not GPTBot.

The AI Indexing Pack includes bot access control that does this verification automatically, so spoofed user agents are treated as ordinary visitors.

Reading the numbers

• Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, bingbot) visiting regularly means you are in those indexes, the prerequisite for citations.
• User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) mean a real person asked an assistant about a page of yours. These are the closest thing to an AI “visit”.
• Training bots (GPTBot, ClaudeBot, CCBot) tell you about future model knowledge, not current answers.
• Requests for llms.txt, .md files and /mcp show agents using the machine-readable layers. On our own site these appear regularly; see our agent-readiness experiment.

Pair it with referrers

Your normal analytics does see one AI signal: people clicking through from an answer. Referrers such as chatgpt.com, perplexity.ai and claude.ai show citations that turned into visits. Logs show the fetch; referrers show the click. Track both monthly, as described in how to get cited by AI assistants.

Turning findings into fixes

403s point at robots and CDN settings; 404s on discovery files point at agent discovery files; agents fetching heavy HTML pages point at Markdown versions. WebAgentScan checks the same access and discovery layers from the outside.

Related reading

• robots.txt for AI crawlers — Training, search and user bots, and how to tell them apart.
• How to get cited by AI assistants — What actually makes ChatGPT, Claude and Perplexity name you.
• Agent discovery files explained — agents.txt, agent cards, server cards and the .well-known tree.
• Markdown for AI agents — Serving a clean text version of every page with content negotiation.
• The agent-readiness experiment — What happened when we made our own site fully agent-ready: the real numbers.
• AI Indexing Pack — The file-drop kit: MCP server, Markdown pages, llms.txt, discovery files and bot access control for any PHP site.
• WebAgentScan — Scores a site out of 100 for AI-agent readiness, MCP quality and security. Paid plans from $9 a month.

Frequently asked questions

Why does Google Analytics not show AI crawlers?

Crawlers and most AI agents do not run JavaScript, so browser-based analytics never records them. Server access logs do.

How do I find GPTBot in my logs?

Search the access log for the GPTBot user agent, for example with grep -i GPTBot access.log, then verify the IPs against OpenAI's published ranges.

How can I tell if a bot is fake?

Check the IP against the provider's published IP list or, where supported, use reverse and forward DNS. Watch for behaviour no real crawler would show.

What does ChatGPT-User in my logs mean?

A person asked ChatGPT about a page of yours and it fetched the page on their behalf.

What should I fix first from the logs?

403 responses to AI search and user agents, then 404s on llms.txt and discovery files, then redirect chains and server errors.