How to audit your site for AI agent readiness and MCP security

The Web Has Changed for Crawlers

AI agents crawl pages every second. ChatGPT, Claude, and Perplexity parse web content to answer user prompts. Most servers hand these crawlers complex HTML, inline scripts, and layout markup. The agent wastes tokens processing DOM nodes instead of extracting facts. When token limits bind, the agent drops your page and moves to a competitor.

Making a site ready for machine consumption requires specific discovery files, structured data, and clean transport layers. It also requires auditing your Model Context Protocol (MCP) endpoints. An unsecured MCP server leaves internal schemas open to prompt injections. You need a clear process to audit, fix, and secure these surface areas.

Step 1: Run an Agent Readiness and Security Audit

Start by evaluating how agents view your origin server. WebAgentScan runs an audit on any domain and generates a score out of 100. It checks seven specific operational categories: agent discovery files, machine-readable content, crawler access control, MCP server quality, agent security posture, link authority, and commerce readiness.

The tool evaluates whether your server exposes llms.txt, agents.txt, and standard .well-known configuration files. It tests if your endpoints respond to content negotiation requests for Markdown. On the security side, it tests your MCP endpoints. It checks auth posture, JSON-RPC responses, exposed tool schemas, and prompt injection vectors. It is the only scanner that scores MCP server security alongside discovery readiness.

WebAgentScan also pulls Domain Rating and referring domain metrics directly from licensed SE Ranking and Ahrefs APIs. It reports these metrics per vendor without blending the numbers. Plans for the hosted SaaS start at $9 per month for 300 scans, $29 per month for 2,000 scans, and $100 per month for unlimited scans with unbadged reports.

Step 2: Fix Discovery Files and Content Negotiation

Once the scanner flags missing files, deploy root-level discovery paths. An agent looks for two primary files before parsing deep links:

• llms.txt: Summarizes site structure, main paths, and core documentation for language models.
• agents.txt: Defines explicit access permissions and endpoints for automated systems.

Your server must also handle content negotiation. When a human visits a page with a browser, the server returns full HTML and CSS. When an agent requests the same URL with an HTTP request header accepting text/markdown, the server should return clean Markdown. This cuts payload size by up to 90 percent. The agent receives plain text facts without processing visual layout code.

Step 3: Secure Your MCP Server Endpoint

Model Context Protocol servers allow external language models to run tool calls against your site data. However, exposing an unauthenticated MCP server creates severe vulnerabilities.

Audit your MCP endpoints for explicit payload schema validation. Verify that tools exposed via JSON-RPC do not allow unauthenticated database queries or systemic prompt overrides. WebAgentScan flags exposed configuration files, loose authorization rules, and vulnerable prompt surfaces. Ensure your server enforces strict authentication headers before serving output schemas to public requests.

Step 4: Deploy Drop-in Packs or Hand-Coded Architecture

Remediating these issues can be handled in two ways, depending on your current stack.

If you already run a site and want to patch it immediately, apply an AI Indexing Pack. This is a one-time $599 download that includes pre-configured llms.txt and agents.txt templates, agent cards, bot access controls, and a working MCP server with output schemas. It installs in about 30 minutes on any server without a monthly recurring software charge.

If you are building a site from scratch, avoid heavy content management systems and visual page builders. BuiltToWinWeb builds custom PHP websites hand-coded from the ground up for a single flat fee. Every build includes a live MCP server, llms.txt, agents.txt, and automated Markdown negotiation built directly into the codebase. You retain full source code ownership with zero monthly platform fees.

Maintain Continuous Readiness

AI agents update their fetching logic regularly. Rescan your origin domain after major deployment updates to ensure your discovery files stay intact and your MCP endpoints remain secured against unauthorized tool access.