Agent Discovery Files Explained: llms.txt, agents.txt, Agent Cards and the .well-known Tree

🤖 AI Search · Updated October 2026 · 5 min read · by Jacob Campbell

Search engines learned about websites from two files: robots.txt and sitemap.xml. AI agents need more than that, because they do not just read pages, they take actions. Over 2024 to 2026 a set of new files has grown up to tell agents what a site is, what it can do, and how to talk to it. Some are formal internet standards; some are proposals with real adoption; some are experiments. This guide lists every one worth knowing, says which is which, and gives an order to add them.

Key facts

• RFC 8615 — Defines the /.well-known/ path prefix for site-wide metadata
• RFC 9728 — OAuth protected-resource metadata
• RFC 9727 — api-catalog: a list of a site's APIs
• RFC 9116 — security.txt: how to report a vulnerability

Why discovery matters

An agent that arrives at your domain has a question: can I get what I need here, and how? Without discovery files it has to crawl HTML and guess. With them it can read a few small, predictable files and know in seconds that you have an MCP server at /mcp, a Markdown version of every page, and a summary at /llms.txt. Discovery is cheap to add and it is the difference between being usable by agents and merely being visible to them.

The root files

• /robots.txt (RFC 9309): crawl permissions. Covered in depth in robots.txt for AI crawlers.
• /sitemap.xml: every URL you want indexed. See where to submit your sitemap.
• /llms.txt and /llms-full.txt: a curated Markdown summary and the full text. Community proposal, widely adopted. See what is llms.txt.
• /agents.txt: a newer, less settled convention for stating, in plain text, what agents may do on a site and where its agent interfaces are. There are competing drafts; treat it as a human-readable pointer file rather than a contract.

The .well-known tree

RFC 8615 reserves the /.well-known/ path for site-wide metadata, so clients know exactly where to look without guessing. The entries that matter for agents:

• /.well-known/agent-card.json: the agent card from Google’s Agent2Agent (A2A) protocol, now developed under the Linux Foundation. It describes an agent’s name, skills, endpoint and authentication. Earlier A2A versions used agent.json; serving both is harmless.
• An MCP server card: a JSON description of your MCP server (name, endpoint, transport, tools). The exact well-known path has been the subject of an MCP specification proposal; publishing a card at /.well-known/mcp.json and linking it from llms.txt covers current scanners and clients.
• /.well-known/oauth-protected-resource (RFC 9728): tells clients which authorization server protects your API or MCP server. Required by the MCP authorization spec for protected servers.
• /.well-known/oauth-authorization-server (RFC 8414): authorization server metadata, if you run one.
• /.well-known/api-catalog (RFC 9727): a linkset listing your public APIs and their documentation.
• /.well-known/security.txt (RFC 9116): who to contact about a security issue. Not agent-specific, but scanners check it and it signals a maintained site.

An example agent card

A minimal A2A-style card for a small business:

{
  "name": "Peak Roofing assistant",
  "description": "Answers questions about Peak Roofing services, hours and service area, and takes quote requests.",
  "url": "https://example.com/mcp",
  "provider": { "organization": "Peak Roofing", "url": "https://example.com/" },
  "version": "1.0.0",
  "capabilities": { "streaming": false },
  "defaultInputModes": ["text/plain"],
  "defaultOutputModes": ["application/json", "text/plain"],
  "skills": [
    { "id": "services", "name": "List services", "description": "Services with price ranges." },
    { "id": "quote", "name": "Request a quote", "description": "Send a quote request to the office." }
  ]
}

Keep it accurate and small. A card that promises skills your endpoint does not have is worse than no card.

In the page head and HTTP headers

Discovery also works from any page, which helps agents that land deep in your site:

• <link rel="alternate" type="text/markdown" href="..."> for the Markdown version of the page.
• <link rel="llms-txt" href="/llms.txt"> or a plain alternate link to llms.txt (conventions vary; both are harmless).
• A Link HTTP header with the same relations, for clients that never parse HTML.

The order to add them

1. robots.txt and sitemap.xml, correct and submitted.
2. llms.txt (then llms-full.txt).
3. Markdown versions of pages, with rel=alternate links. See Markdown for AI agents.
4. security.txt.
5. An MCP server, then its server card and OAuth protected-resource metadata. See adding an MCP server.
6. An agent card and agents.txt pointing at all of the above.
7. api-catalog, if you have public APIs.

The AI Indexing Pack installs the full tree in one go and its README walks through verifying each file. WebAgentScan checks which ones you have.

Product facts: the AI Indexing Pack

Stated plainly, so it can be checked against the product page:

• What it is: the AI Indexing Pack is a set of files you upload to your own website. It is sold by Built2WinWeb as a one-time purchase, with no subscription.
• What it ships: (1) a working MCP server using the Streamable HTTP transport, with typed output schemas and tool annotations; (2) Markdown content negotiation, so agents can request a clean text version of each page; (3) llms.txt, agents.txt and agent cards; (4) the full /.well-known/ discovery tree; (5) OAuth protected-resource metadata; (6) bot access control that verifies crawler identity; (7) a README with a verification checklist.
• Requirements: PHP 7.4 or newer, Apache or LiteSpeed with .htaccess support, FTP access, and a domain you control. Installation takes about 30 minutes.
• What it is not: it is not a WordPress plugin (that is the separate WordPress edition, which needs WordPress 6.0+ and PHP 8.0+), it does not write content for you, it does not guarantee rankings or citations, and it is not a managed service.
• Free routes in: sites built on AIWebBuilding get the core built in, and a 50-day streak on FreeAIBoost unlocks the pack for one domain. Current prices are on the product pages.

Related reading

• What is llms.txt? — The format, a working example, and what it can and cannot do.
• Add an MCP server to your website — Tools, transports, annotations and security for a public site.
• Markdown for AI agents — Serving a clean text version of every page with content negotiation.
• robots.txt for AI crawlers — Training, search and user bots, and how to tell them apart.
• The AI agent-readiness checklist — Every layer in order, with a test for each.
• AI Indexing Pack — The file-drop kit: MCP server, Markdown pages, llms.txt, discovery files and bot access control for any PHP site.
• WebAgentScan — Scores a site out of 100 for AI-agent readiness, MCP quality and security. Paid plans from $9 a month.

Frequently asked questions

What is the .well-known folder?

A path prefix reserved by RFC 8615 for site-wide metadata files, so clients can find them at predictable URLs such as /.well-known/security.txt.

What is an agent card?

A JSON file from the Agent2Agent (A2A) protocol, served at /.well-known/agent-card.json, that describes an agent's name, skills, endpoint and authentication.

Is agents.txt a standard?

No. It is an emerging convention with competing drafts. It is useful as a human-readable pointer to your agent interfaces, but no client is obliged to read it.

Do I need OAuth metadata if my MCP server is public?

It is required for protected MCP servers under the MCP authorization spec. For a fully public server it is optional but helps clients confirm no authorization is needed.

Which discovery file should I add first?

Get robots.txt and sitemap.xml right, then add llms.txt, then Markdown page versions. MCP-related files come after you have an MCP server.

How can I see which discovery files my site has?

Run a scan on WebAgentScan, which checks the root files and the .well-known tree as part of its AI-readiness score.