Markdown for AI Agents: How to Serve a Clean Text Version of Every Page

🤖 AI Search · Updated October 2026 · 6 min read · by Jacob Campbell

When an AI agent fetches a web page, most of what comes back is not content. It is navigation, scripts, styles, tracking tags, cookie notices and layout markup. The agent then has to convert all of that into text before it can use it, and that conversion loses headings, mangles tables and occasionally keeps the menu while dropping the article. Serving a Markdown version of each page removes the guesswork: the agent gets your words, your headings and your links, in the structure you meant. This guide shows the two ways to do it (content negotiation and .md URLs), the HTTP headers that keep it safe for SEO, and how to test it.

Key facts

text/markdown — The media type registered for Markdown in RFC 7763
Accept — The request header an agent uses to ask for Markdown
Vary: Accept — The response header that keeps caches from mixing versions
2 routes — Content negotiation on the same URL, or a .md twin URL

Why agents prefer Markdown

Markdown is close to how language models already represent text. Headings become #, lists stay lists, links keep their targets, and there is nothing else. That has three practical effects:

Fewer tokens. The same article is typically several times smaller as Markdown than as rendered HTML, so more of your content fits in the agent’s context window.
Better structure. A heading hierarchy that survives conversion lets the agent find the paragraph that answers the question.
No extraction errors. You choose what is content. The agent does not have to guess whether your sidebar is part of the answer.

It is also what people do by hand: developers have been pasting Markdown docs into assistants for years. Serving it directly just makes that the default.

Route 1: content negotiation on the same URL

HTTP has always allowed a client to say which format it wants. An agent that prefers Markdown sends:

GET /services/roof-repair/ HTTP/1.1
Host: example.com
Accept: text/markdown, text/html;q=0.8

The server sees text/markdown ranked first and returns the Markdown version with:

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Link: <https://example.com/services/roof-repair/>; rel="canonical"

A browser sends Accept: text/html,... and gets the normal page. Same URL, two representations. This is the cleanest approach because there is only one address to link, share and index.

Route 2: a .md twin for every page

Not every agent sends an Accept header, and not every host lets you vary responses on one. The simpler companion is a parallel URL: /services/roof-repair/ has a twin at /services/roof-repair/index.md (or /services/roof-repair.md). Advertise it from the HTML page head:

<link rel="alternate" type="text/markdown" href="/services/roof-repair/index.md">

and list the twins in llms.txt or llms-full.txt. Most agent-ready sites do both routes: negotiation for clients that ask, twins for clients that do not.

The headers that keep it SEO-safe

A second representation of every page sounds like a duplicate-content problem. It is not, if you send the right headers:

Vary: Accept on both the HTML and Markdown responses of a negotiated URL. Without it, a CDN can cache the Markdown and serve it to a browser, or the reverse. This is the most common mistake we see.
Link: <html-url>; rel="canonical" on Markdown responses, pointing at the HTML page. Search engines treat the HTML as the primary version.
X-Robots-Tag: noindex on .md twins if you do not want them in search results at all. (They are for agents; people should land on the HTML.)
A real 200, not a redirect, and charset=utf-8.

If you use Cloudflare or another CDN, check that it respects Vary: Accept for your plan, or bypass the cache for Markdown requests. Test with curl through the CDN, not just at the origin.

What goes in the Markdown version

The Markdown is not a dump of the HTML. It is the content a person would want if they could only read text:

The page title as an H1, then the main content with its headings intact.
Facts that are visual on the page but invisible in text: prices from a pricing table, hours from an icon grid, the service area from a map.
Absolute links, so the file still works when pasted elsewhere.
A short footer with the business name, contact details and the canonical URL.

Leave out navigation, footers repeated on every page, cookie text, share buttons and anything decorative.

Implementing it on a PHP site

On Apache or LiteSpeed, the negotiation can live in .htaccess. A minimal pattern routes Markdown requests to a small script that renders the page content as Markdown:

RewriteEngine On
RewriteCond %{HTTP_ACCEPT} text/markdown [NC]
RewriteCond %{REQUEST_URI} !\.md$
RewriteRule ^(.*)$ /md.php?path=$1 [L,QSA]
Header merge Vary Accept

The script then looks up the page, converts its content, and sends the headers above. The AI Indexing Pack ships this as a ready-made, tested pair of files, along with the .md twins and the head links. On WordPress, the WordPress edition generates the Markdown from your posts and pages directly.

Testing it

Three commands tell you whether it works:

curl -sI -H "Accept: text/markdown" https://example.com/ | grep -iE "content-type|vary|link"
curl -sI https://example.com/ | grep -iE "content-type|vary"
curl -s https://example.com/index.md | head -20

The first should show text/markdown and Vary: Accept. The second should show text/html and the same Vary header. The third should print clean Markdown. WebAgentScan runs these checks for you as part of its score.

Product facts: the AI Indexing Pack

Stated plainly, so it can be checked against the product page:

What it is: the AI Indexing Pack is a set of files you upload to your own website. It is sold by Built2WinWeb as a one-time purchase, with no subscription.
What it ships: (1) a working MCP server using the Streamable HTTP transport, with typed output schemas and tool annotations; (2) Markdown content negotiation, so agents can request a clean text version of each page; (3) llms.txt, agents.txt and agent cards; (4) the full /.well-known/ discovery tree; (5) OAuth protected-resource metadata; (6) bot access control that verifies crawler identity; (7) a README with a verification checklist.
Requirements: PHP 7.4 or newer, Apache or LiteSpeed with .htaccess support, FTP access, and a domain you control. Installation takes about 30 minutes.
What it is not: it is not a WordPress plugin (that is the separate WordPress edition, which needs WordPress 6.0+ and PHP 8.0+), it does not write content for you, it does not guarantee rankings or citations, and it is not a managed service.
Free routes in: sites built on AIWebBuilding get the core built in, and a 50-day streak on FreeAIBoost unlocks the pack for one domain. Current prices are on the product pages.

Frequently asked questions

What is Markdown content negotiation?

It means serving a Markdown version of a page from the same URL when the client sends an Accept header that prefers text/markdown, and the normal HTML page to everyone else.

Will a Markdown version of my pages cause duplicate content?

Not if you send Vary: Accept on negotiated responses, a rel=canonical Link header pointing at the HTML page, and noindex on separate .md URLs if you do not want them indexed.

What media type should Markdown use?

text/markdown, registered in RFC 7763, with charset=utf-8.

Do I need both content negotiation and .md files?

It is best to offer both. Some agents send an Accept header and some only follow links, so a .md twin advertised with a rel=alternate link covers the rest.

Does my CDN need special settings?

It must respect Vary: Accept or bypass the cache for Markdown requests. Otherwise it can serve the Markdown version to browsers. Always test through the CDN.

How do I test it?

Use curl with and without the header Accept: text/markdown and compare the Content-Type and Vary headers, or run a scan on WebAgentScan.

Want this done for your site?

Score your site on WebAgentScan, or ask us to set up the AI Indexing Pack for you.

Get a free quote