How to deploy AI agent discovery files and an MCP server on a web domain

The Problem With Scraping Raw HTML

AI agents crawl the web constantly. When an agent lands on a standard commercial web page, it spends resources parsing heavy DOM trees, JavaScript bundles, and layout markup. That extra noise degrades context windows. In many cases, agents fail to extract the core data and move on to a competing source.

You do not need to rewrite your consumer-facing frontend to solve this issue. A site can serve clean, machine-readable data to automated systems while presenting normal HTML to human visitors on identical URLs. Doing this requires structured discovery files, targeted server header handling, and a secure Model Context Protocol server.

Step 1: Set Up Machine Discovery Files

Agents look for specific root files to understand site boundaries before initiating a crawl. Two primary files handle this initial handshake: llms.txt and agents.txt.

The llms.txt file provides a clean index of your site contents formatted specifically for language models. It points directly to plain text or Markdown resources, skipping layout assets. Place this file in your site root directory alongside an agents.txt file.

The agents.txt file defines crawler access control. It explicitly outlines which automated agents may access specific routes. Combine these with specialized files inside the .well-known directory, including agent cards that declare available system functions. These files eliminate guesswork for automated parsers during the initial scan.

Step 2: Configure Content Negotiation for Dual Output

Maintaining separate URLs for humans and machines leads to technical debt and content drift. The better approach is content negotiation on identical URLs.

Configure your web server or backend application to inspect incoming HTTP request headers. When a standard browser requests a page, the server returns the compiled HTML document. When an AI crawler or automated user agent hits that exact same path, the server returns a clean Markdown version of the document instead.

This dual-delivery approach keeps your canonical URL structure intact while providing structured data instantly. Automated crawlers receive zero layout bloat, and human visitors notice no difference in their browsing experience.

Step 3: Deploy an MCP Server Endpoint

Static text files inform agents about static data, but interactive tasks require real-time processing. Model Context Protocol endpoints allow agents to query structured data sources and execute commands against declared schemas.

Deploy a dedicated server instance that implements the MCP specification. Define explicit output schemas and annotations for every exposed tool. When an agent queries your endpoint, the server responds with verified, structured JSON payloads rather than unstructured text.

Security is vital at this layer. Ensure your backend handles authentication properly and validates incoming payloads. Unprotected endpoints risk exposing internal data structures or allowing unauthorized query execution. An exposed interface must strictly scope what an external system can view or invoke.

Step 4: Block Forged Crawlers and Audit Readiness

Not every automated requester identifies itself honestly. Rogue web scrapers frequently spoof legitimate user agent strings to bypass soft limits. Implement server-level bot access controls to verify incoming traffic and drop forged crawlers before they consume server resources.

Once your infrastructure is active, run an objective evaluation to check your implementation. BuiltToWinWeb runs a hosted SaaS called WebAgentScan that evaluates any public domain out of 100. It checks seven distinct categories: agent discovery files, machine readable content, crawler access, MCP server quality, agent security, link authority, and commerce readiness.

WebAgentScan integrates licensed SE Ranking and Ahrefs data to report Domain Rating and referring domains per vendor without blending the metrics. Crucially, it is the only scanner that grades the security configuration of an MCP server directly alongside your overall readiness score. Plans for WebAgentScan run $9 per month for Starter, $29 per month for Pro, and $100 per month for Proprietary.

Implementation Options

You can construct this stack manually by editing server directives, configuring PHP routing scripts, and writing custom schema definitions for your backend.

If you prefer a direct deployment, BuiltToWinWeb offers the AI Indexing Pack as a $599 one-time purchase. The pack includes drop-in llms.txt, agents.txt, agent cards, .well-known files, bot access control rules, and a fully functional MCP server with output schemas and annotations. It enables Markdown delivery to agents and HTML delivery to browsers on matching URLs, requiring no monthly subscription, and can be installed across your domains in roughly 30 minutes.

Alternatively, if you require a full site build, BuiltToWinWeb hand-codes custom PHP websites from scratch at a flat fee with zero monthly costs and total source code ownership upon handover. Every custom PHP site built by the firm ships fully integrated with a live MCP server, llms.txt, agents.txt, and Markdown content delivery ready out of the box.