How AI Assistants Read a Web Page: Fetching, Rendering and Extraction Explained

🔎 AI Search · Updated October 2026 · 10 min read · by Jacob Campbell

When an AI assistant cites your page, it has either looked it up in a search index built earlier by a crawler, or fetched it live because a user’s question needed it, and in most cases it read the HTML your server sent without running your JavaScript. Google is the main exception: its AI Overviews, AI Mode and Gemini grounding draw on the Google Search index, which Googlebot builds by rendering pages in Chromium. This guide walks through each step of that pipeline (fetching, rendering, extraction), what commonly breaks it, and the tests you can run in five minutes to see your page the way an assistant does.

Key facts

• 2 routes — A pre-built search index, or a live fetch triggered by the user’s question
• No JS — Anthropic’s web fetch tool does not support pages rendered with JavaScript, per its docs
• 80% — Token reduction Cloudflare measured converting one of its blog posts from HTML to Markdown
• July 1, 2025 — Date Cloudflare began asking every new domain whether to allow AI crawlers

Two routes to your page: the index and the live fetch

Every AI assistant that cites the web gets your content by one of two routes: from a search index that a crawler built before the question was asked, or from a live fetch made while it is answering.

• Google AI Overviews and AI Mode use the Google Search index. Google says a page must be indexed and eligible to show a snippet to appear as a supporting link, and that there are “no additional technical requirements” beyond that. Both features may use “query fan-out”, issuing several related searches to find supporting pages (Google: AI features and your website). Gemini Apps also ground answers in the Search index, which is what the Google-Extended token controls.
• ChatGPT search rewrites the user’s question into one or more search queries, sometimes sends them to partner search providers (OpenAI names Microsoft), and shows citations. To be included, OpenAI says your site must allow OAI-SearchBot and must not block its published IP addresses (OpenAI: ChatGPT search). When ChatGPT opens a specific page for a user, the request comes from ChatGPT-User.
• Perplexity builds its own index with PerplexityBot and fetches pages live with Perplexity-User.
• Claude uses Claude-SearchBot for search indexing and Claude-User when a user’s question leads it to a page.
• Microsoft Copilot: Microsoft has not documented a separate AI crawler, and Bing Webmaster Tools reports how often your pages are cited in Copilot, so the practical requirement is that bingbot can crawl and index you.

This matters for debugging. If an assistant describes your page out of date, it is probably reading an index copy; if it says it could not access the page, the live fetch failed. The full list of user agents is in the AI crawler user-agent list.

Does the assistant run your JavaScript?

Assume it does not: apart from Google and Apple, the AI fetchers that have been tested or documented read the HTML your server returns and do not execute JavaScript.

The evidence, from the most authoritative source down:

• Anthropic states in its documentation that its web fetch tool “currently does not support websites dynamically rendered with JavaScript”, and points developers who need rendering to a separate browser tool (Claude web fetch tool).
• Google renders. Its documentation describes three phases (crawling, rendering, indexing) and says rendering uses “an evergreen version of Chromium”. Pages wait in a render queue, usually seconds but sometimes longer. Google also warns that “not all bots can run JavaScript” and recommends server-side rendering or pre-rendering (JavaScript SEO basics).
• Apple says Applebot “may render the content of your website within a browser” (About Applebot).
• Everyone else: an analysis of crawler traffic published by Vercel with MERJ in December 2024 found that “none of the major AI crawlers currently render JavaScript”, naming OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, meta-externalagent, Bytespider and PerplexityBot. The bots did download JavaScript files (11.5% of ChatGPT’s fetches and 23.8% of Claude’s) but did not execute them (Vercel: The rise of the AI crawler). That data is from 2024, and operators can change their fetchers without notice, but no operator has since documented that its fetcher renders JavaScript.

Browsing agents are a separate case. Tools that drive a real browser on a user’s behalf can run JavaScript, but they are used for tasks like filling in forms, not for the fast fetch behind a typical cited answer. Design for the plain fetch.

Practical rule: everything you want quoted (headings, body text, prices, opening hours, FAQ answers, links) must be present in the initial HTML response. Content injected by a client-side framework after load, text inside tabs that are populated by JavaScript, and “load more” sections that fetch on click are invisible to most assistants.

What gets extracted from the HTML

After fetching, the assistant reduces your page to text, and only that text, possibly truncated, reaches the model.

Anthropic’s web fetch tool is a documented example of the process: HTML and plain-text responses are returned to the model as plain text, PDFs are passed as documents, and a max_content_tokens setting truncates anything longer than the limit. Anthropic estimates an average 10 kB page at about 2,500 tokens and a 100 kB documentation page at about 25,000. Other operators do not document their extraction, but the shape is the same everywhere: markup is stripped, and the more of your HTML is navigation, scripts, inline styles and repeated boilerplate, the less of the token budget is left for your content.

What survives extraction best:

• Headings that describe their section (<h2>, <h3>), because they mark where an answer starts.
• The answer near the top, in a self-contained first sentence, in case the page is truncated.
• Real lists and tables (<ul>, <ol>, <table>) rather than grids of <div>s styled to look like a table, which flatten into an unlabelled run of words.
• Content in <main> and <article>, which separates it from headers, footers and sidebars.
• Text, not text in images. A price list as a JPEG is not a price list to a text extractor.

Structured data in JSON-LD is part of the HTML too, and Google documents its use for rich results, but no assistant operator documents how much weight its extractor gives it. Write the facts in visible text first; mark them up second.

Why server-rendered HTML, Markdown and fast responses matter

The pages assistants read most reliably are the ones that return complete, lightweight HTML quickly, to any client, on the first request.

• Server-rendered HTML. Server-side rendering, static generation or pre-rendering all put the content in the first response. If you use React, Vue or similar, make sure the framework renders on the server for public pages; check by viewing source (see the tests below).
• A Markdown version. Markdown carries the same headings, lists and links with far fewer tokens. Cloudflare measured one of its own blog posts at 16,180 tokens as HTML and 3,150 as Markdown, an 80% reduction, and its Markdown for Agents feature, launched in February 2026, serves Markdown when a client sends Accept: text/markdown. Cloudflare reports that coding agents including Claude Code and OpenCode already send that header (Cloudflare: Markdown for Agents). How to set it up on your own server: Markdown for AI agents.
• Fast, stable responses. A live fetch happens while a user waits for an answer. None of the operators we checked publish a fetch timeout, so there is no safe number to aim for beyond “as fast as you can”; a page that takes several seconds to return its first byte is at risk of being skipped. The same server work that improves Core Web Vitals (caching, a fast host, no blocking server-side calls) helps here.
• Correct status codes. Return 200 for real pages, 301 for moved ones and 404 or 410 for gone ones. The Vercel/MERJ data found ChatGPT’s and Claude’s crawlers spending about 34% of fetches on 404 pages, which suggests they request stale URLs; redirects keep those requests useful.

What blocks AI fetchers

Most failed AI fetches are caused by something the site owner set up for other reasons: bot protection, consent walls, client-side rendering or login requirements.

• Bot protection and WAF rules. Cloudflare began asking every new domain whether to allow AI crawlers on July 1, 2025, and many sites answered no without realising that also covers search crawlers. Other CDNs and security plugins block unfamiliar user agents, rate-limit by IP, or serve JavaScript challenges that a non-rendering fetcher can never pass. OpenAI explicitly says your host must allow traffic from its published IP addresses.
• robots.txt. A broad Disallow aimed at training bots can catch search crawlers too. See robots.txt for AI crawlers.
• Cookie and consent walls. If the consent banner is rendered server-side in place of the content, or the content is only inserted after consent is given, the fetcher sees the banner. Overlay banners that sit on top of content already in the HTML are fine.
• Client-side rendering. An empty <div id="root"></div> plus a script bundle is an empty page to most assistants.
• Paywalls, logins and geo-blocks. Content behind authentication is not fetched. Region blocks can fail fetchers whose servers are in another country.
• Slow time to first byte. See above: a fetch made while the user waits has little patience.
• Snippet and indexing controls. For Google, noindex, nosnippet, data-nosnippet and max-snippet all limit what AI features can show, by Google’s own documentation. A noindex in the initial HTML may also stop Google rendering the page at all. Search Console’s Settings > Search generative AI control, available to all sites since August 31, 2026, can exclude a whole site from AI Overviews and AI Mode; if your pages never appear there, check nobody switched it off.

Five tests to see your page as an assistant does

You can check most of the above with a terminal and a browser in a few minutes.

1. Fetch the page with an AI user agent and time it.

UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot'
curl -sS -A "$UA" -o page.html \
  -w 'status=%{http_code} ttfb=%{time_starttransfer}s total=%{time_total}s bytes=%{size_download}\n' \
  https://www.example.com/your-page/

You want status=200, a small TTFB and a byte count that looks like a real page rather than a challenge screen.

2. Check the content is in the raw HTML. Pick a sentence from the middle of your main content and search for it:

grep -c "a distinctive sentence from your page" page.html

A count of 0 means the text is added by JavaScript, or the response was a block page. Open page.html to see which.

3. Compare view-source with the rendered page. In Chrome, View Page Source (Ctrl+U) shows what the server sent; the Elements panel in DevTools shows the DOM after JavaScript. Or open the DevTools command menu (Ctrl+Shift+P), run “Disable JavaScript” and reload: what you see is roughly what a non-rendering fetcher gets. For Google specifically, Search Console’s URL Inspection tool shows the HTML Googlebot rendered.

4. Ask for Markdown.

curl -sS -D - -o /dev/null -H 'Accept: text/markdown' https://www.example.com/your-page/

If the Content-Type response header says text/markdown, your server negotiates Markdown.

5. Read robots.txt as the bot would. Fetch https://www.example.com/robots.txt and check which group applies to each AI user agent you care about.

One caveat on test 1: a request from your own machine with a bot user agent is not a real bot request. Some bot-protection services block a request that claims to be a known bot but comes from an IP the bot does not use, so you may see a block that the real bot would not, or, more often, pass a check that the real bot fails on IP reputation. Your server logs are the ground truth: look for real requests from these user agents and their status codes (AI crawlers in server logs).

Limits: what you cannot control

Making a page easy to read does not make an assistant read it, cite it or quote it accurately.

• Which pages an assistant searches for, opens and cites is decided by its own retrieval and ranking systems, which no operator documents in detail. Google says only that standard Search eligibility applies to AI features.
• Index copies go stale. An assistant can quote last month’s price from its index even after your page changed; keep facts consistent across your site and your structured data so old and new copies agree.
• Operators change fetchers without announcement. The JavaScript findings above are a 2024 measurement plus current documentation; re-test once in a while.
• Not everything is worth unblocking. You may have good reasons to keep a members area, a booking flow or a pricing tool away from bots. The goal is to make the public, quotable pages readable, not to remove all protection.

For the content side of being cited, see how to get cited by AI assistants; for measuring whether it worked, see measuring traffic from AI assistants.

Product facts: the AI Indexing Pack

Stated plainly, so it can be checked against the product page:

• What it is: the AI Indexing Pack is a set of files you upload to your own website. It is sold by Built2WinWeb as a one-time purchase, with no subscription.
• What it ships: (1) a working MCP server using the Streamable HTTP transport, with typed output schemas and tool annotations; (2) Markdown content negotiation, so agents can request a clean text version of each page; (3) llms.txt, agents.txt and agent cards; (4) the full /.well-known/ discovery tree; (5) OAuth protected-resource metadata; (6) bot access control that verifies crawler identity; (7) a README with a verification checklist.
• Requirements: PHP 7.4 or newer, Apache or LiteSpeed with .htaccess support, FTP access, and a domain you control. Installation takes about 30 minutes.
• What it is not: it is not a WordPress plugin (that is the separate WordPress edition, which needs WordPress 6.0+ and PHP 8.0+), it does not write content for you, it does not guarantee rankings or citations, and it is not a managed service.
• Free routes in: sites built on AIWebBuilding get the core built in, and a 50-day streak on FreeAIBoost unlocks the pack for one domain. Current prices are on the product pages.

Related reading

• AI crawler user-agent list (2026) — Every documented AI bot, its purpose, robots.txt behaviour and how to verify it.
• Markdown for AI agents — Serve a clean Markdown version of every page with content negotiation.
• robots.txt for AI crawlers — Which bots to allow, which to block, and the trade-offs of each choice.
• Find AI crawlers in your server logs — Which bots visit, what they fetch and what they get back.
• How to get cited by AI assistants — The content and technical work that makes a page worth citing.
• Measure traffic from AI assistants — Referrers, utm_source=chatgpt.com, GA4 channel groups and citation reports.
• AI Indexing Pack — The file-drop kit: MCP server, Markdown pages, llms.txt, discovery files and bot access control for any PHP site.

Frequently asked questions

Do ChatGPT, Claude and Perplexity run JavaScript when they read a page?

Generally no. Anthropic documents that its web fetch tool does not support JavaScript-rendered sites, and a 2024 Vercel and MERJ analysis found that none of the major AI crawlers, including OpenAI’s, Anthropic’s and Perplexity’s, executed JavaScript.

Does Google render JavaScript for AI Overviews?

AI Overviews and AI Mode draw on the Google Search index, and Googlebot renders pages with an evergreen version of Chromium before indexing. Google still recommends server-side rendering because rendering can be delayed and not all bots run JavaScript.

How can I see my page the way an AI assistant sees it?

Fetch it with curl using an AI user agent and check that your main text appears in the raw HTML, or disable JavaScript in Chrome DevTools and reload. If the content disappears, most AI fetchers cannot see it.

Can a cookie banner stop AI assistants reading my site?

Yes, if the banner replaces the content in the HTML or the content only loads after consent. A banner that overlays content already present in the server-sent HTML does not hide that content from a text extractor.

Why serve a Markdown version of pages?

Markdown keeps headings, lists and links while using far fewer tokens than HTML; Cloudflare measured an 80% reduction on one of its blog posts. Some agents, such as Claude Code, already request it with an Accept: text/markdown header.