Serving Markdown to AI agents with PHP content negotiation

The problem with serving HTML DOM trees to bots

When standard web crawlers visit your site, they execute JavaScript and process HTML layouts. AI agents from OpenAI, Anthropic, and Perplexity parse web pages differently. They discard layout styling, navigation menus, and script tags to extract primary text content. Handing these bots a multi-megabyte HTML tree wastes bandwidth and increases token consumption. It also increases the chances of parser errors when an LLM tries to reconstruct your core message from nested DOM structures.

To fix this, web developers often maintain separate static text files. Maintaining duplicate markdown files for hundreds of pages creates sync debt. If you update a product page in your database, your markdown files immediately drift out of sync. The solution is native content negotiation directly inside your application router. By evaluating HTTP request headers, a plain PHP application can serve standard HTML to human visitors while streaming clean Markdown to scrapers at the exact same URL endpoint.

Implementing user agent detection PHP and header parsing

HTTP content negotiation relies on two primary request attributes: the Accept header and the User-Agent string. When an LLM crawler or customized agent fetches a page, it usually specifies its preferred MIME type or identifies itself via standard headers. You can evaluate these values early in your request pipeline using native PHP arrays.

The execution flow begins by checking if the incoming request asks explicitly for Markdown or originates from a known crawler string. Here is how simple user agent detection PHP logic operates before rendering views:

You read $_SERVER['HTTP_ACCEPT'] to check for MIME types like text/markdown or text/x-markdown. Simultaneously, you inspect $_SERVER['HTTP_USER_AGENT'] against a list of common AI agent user agents. If either condition evaluates to true, your backend skips template rendering entirely and switches execution paths to output plain text structured with standard H1, H2, and bullet formatting.

This approach gives you a clean way to serve markdown to ai agents without maintaining duplicate file directories or managing complicated static asset pipelines.

Configuring PHP content negotiation for dynamic pages

To execute php content negotiation reliably, your code must set explicit HTTP response headers before emitting any output text. Setting the correct response headers prevents downstream CDN caches from serving Markdown responses to standard browser users.

Your backend logic should output two critical headers:

• Content-Type: text/markdown; charset=utf-8
• Vary: Accept, User-Agent

The Vary header tells reverse proxies, edge workers, and browser caches that the response body changes depending on who makes the request. Without this header, a CDN might cache a Markdown payload requested by an agent and serve plain text to the next human user who loads the page in Chrome.

When building custom PHP systems, integrating header evaluation into your primary router ensures every content route handles request switching transparently. As described in our guide on how to deploy AI agent discovery files and an MCP server on a web domain, clean protocol handling forms the foundation of machine-accessible site design.

Generating a dynamic llms txt file and structured content

Once your header checks are in place, you can build a lightweight layout engine that converts database records into Markdown strings. Instead of passing query results into an HTML view template, pass them to a plain-text formatter. Convert standard HTML fields to raw text using helper functions or basic string transformations.

This same dynamic architecture allows you to generate a dynamic llms txt index at the site root. Rather than editing a static file every time you publish a post, query your database for published URLs, titles, and summaries, then stream them out as an index document under the standard file path.

Dynamic output simplifies downstream integrations. When passing enriched page content or lead capture details off to external processing queues, clean text streams avoid parsing overhead. Teams setting up background worker tasks—like structuring CRM webhooks to enrich inbound leads before sales assignment—rely on structured data schemas to eliminate payload junk before execution.

Auditing and testing your agent endpoints

After implementing content negotiation, verify that your backend behaves correctly under different request headers. Standard browser tools hide header overrides, so command-line testing with cURL provides the fastest feedback loop.

Run cURL commands specifying an explicit accept header to confirm the server returns Markdown:

curl -H "Accept: text/markdown" https://example.com/page

Next, simulate an AI agent crawler by setting a custom user-agent string:

curl -A "GPTBot/1.0" https://example.com/page

Both calls should return plain Markdown and set Content-Type: text/markdown. Loading the exact same URL without custom headers in a standard browser must continue to serve fully rendered HTML.

To verify overall site readiness, run automated checks against your endpoints. In our walkthrough on how to resolve site readiness issues and secure MCP endpoints, we examine how hosted evaluation platforms like WebAgentScan scan domains across seven graded areas to ensure discovery files, headers, and security rules align with automated parser expectations.