I Made My Site Fully Agent-Readable. Here Are the Real Numbers.

🤖 Agent readiness · Updated August 2026 · 9 min read · by Jacob Campbell

Over the past few months I turned built2winweb.com into about as agent-readable a site as a solo developer can build: an MCP server, an A2A agent card, an NLWeb /ask endpoint, ACP, UCP, agents.txt, llms.txt, an RSL licence, Markdown content negotiation on every page, and IndexNow wired to the CDN. Then I sat down and actually measured it. This is the honest version — including the parts that did not work, the two Cloudflare defaults that were fighting me, and the number that matters more than all the protocol plumbing combined.

By the numbers

• 209,600 — URLs pushed to IndexNow
• 1,319 — Pages actually indexed by Bing
• 28,685 — URLs in the sitemap
• 16 — Referring domains — the real ceiling

What I actually built

The full surface is documented on the agent-ready page, but briefly: an MCP server at /mcp exposing seven tools, a SEP-2127 server card, an A2A agent card with the AP2 merchant extension, an ACP descriptor, a UCP commerce profile, per-path agent permissions, an /ask endpoint running BM25 over the site with Schema.org-typed responses, llms.txt and llms-full.txt, AGENTS.md, a Markdown sitemap, and an RSL 1.0 licence referenced from robots.txt that explicitly permits search indexing, RAG and model training.

Every one of those is live, public and needs no API key. None of it uses a framework — it is plain PHP on shared hosting. If you want to see it, call it.

What happened: the numbers

Here is where honesty is more useful than a case study. Across roughly three months:

• 209,600 URLs submitted to IndexNow
• 1,319 pages indexed in Bing
• 28,685 URLs in the sitemap
• 111 impressions, 0 clicks in Bing search
• 16 referring domains

Seznam tells the same story from another angle: 3,390 URLs discovered, 86 kept in its crawler database, 19 indexed. That is 0.6% of what it found.

So: I notified search engines about my pages roughly seven times over, and about 4.5% of the site is indexed. Notification was never the constraint.

Cloudflare Crawler Hints was firing 4,900 URLs every 18 hours

This is the part I would not have caught without opening Bing Webmaster Tools. Every single one of those 209,600 IndexNow submissions came from Cloudflare Crawler Hints, which I had enabled and forgotten. On a 28,685-URL site it was pushing 4,900 URLs every 18 hours — including pages I had deliberately marked noindex, and at least one .well-known challenge file that returns 404.

Bing noticed. Its top recommendation on my dashboard read: "Avoid IndexNow Batch Mode to prevent excessive server load and potential indexing delays." I had managed to turn a helpful protocol into a self-inflicted problem by leaving a toggle on. IndexNow is for content that changed. Firing your whole site at it repeatedly is how a key gets deprioritised.

My CDN was contradicting my own robots.txt

The second surprise was worse, because it worked directly against the thing I was trying to do. My robots.txt spends about 200 lines explicitly welcoming AI crawlers — GPTBot, ClaudeBot, PerplexityBot, CCBot and the rest — with Content-Signal: search=yes, ai-input=yes, ai-train=yes.

Above all of that, Cloudflare was injecting a managed block that said the opposite:

# BEGIN Cloudflare Managed content
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# END Cloudflare Managed Content

It also declared ai-train=no where mine said yes. Under RFC 9309 duplicate user-agent groups get merged and Allow: / wins a tie, so well-behaved crawlers probably still fetched — but I was relying on tie-breaking to undo a directive I never wrote. Check your rendered robots.txt, not your source file. Your CDN may be editing it.

Discovery and citation are different problems

Here is the conclusion I did not want but the data insists on. Every protocol I implemented solves discovery — helping a machine find and parse what exists. None of them solve authority — giving anyone a reason to cite it.

With 16 referring domains, Bing is choosing not to index most of my site, and no amount of notification changes that decision. Bing said so directly in its second recommendation: "Your site does not have enough inbound links from high quality domains." Two search engines, zero crawl errors, zero guidelines violations, and still almost nothing indexed. That is not a technical failure. It is a judgment about whether the content is worth storing.

If you are considering this work as an SEO play, do not. It is not one.

The bug that was hiding underneath all of it

While auditing, I found something more embarrassing and more consequential than any protocol detail. Thirteen of my article URLs were serving the wrong article.

My rewrite rules pointed several slugs at positional anchors — blog-card-1, blog-card-3 — that referred to slots on the blog index rather than to specific posts. When the index got reordered, the slugs silently detached from their content. A URL about migrating from Shopify was serving an article about critical CSS. One of them was in my sitemap.

Every anchor I had named descriptively was still correct. Not one had drifted. The lesson is boring and universal: name things after what they are, not where they sit. And an index full of pages that do not deliver what their title promises is a very good explanation for a 0% click-through rate.

What I would tell you to do

If you want to make a site agent-readable, the order that actually matters:

1. Fix what is broken first. I found mismatched routes, 2,020 stale 404s from apostrophes in city slugs, and a CSS rule that had been rendering a page white-on-white. Those cost more than any missing manifest.
2. Publish llms.txt and an /ask-style endpoint. Cheap, genuinely useful, and lets an assistant get a cited answer instead of guessing.
3. Check what your CDN is doing on your behalf. Both of my biggest problems were defaults I never chose.
4. Then earn links. This is the slow one, and it is the one that decides everything else.

So was it worth doing?

Yes — but not for the reason I started. It bought precision, not traffic. When an assistant is asked what a custom site costs, it can now fetch a typed answer with a citation instead of paraphrasing marketing copy. Nothing is generated on my origin, so nothing about my pricing gets hallucinated. That is worth something on its own, and it is a reasonable bet on how people will find things next.

What it is not is a shortcut. There is no manifest you can publish that substitutes for being worth citing. I would rather tell you that plainly than sell you the version where the graph goes up.

If you want a site built this way

I hand-code every site I build in PHP — no WordPress, no page builders, no monthly fees — and the agent surface above is the same kind of work. If you want a fast site that both people and machines can actually read, get a free quote and tell me what you need.

Related reading

• Every endpoint, documented — The full agent surface with live examples.
• Where to submit your sitemap — IndexNow and every search engine that matters.
• The backlink gap — The constraint this whole post runs into.

Frequently asked questions

Did adding MCP, A2A and llms.txt improve search rankings?

No. Across three months the site went from a handful of indexed pages to 1,319 out of 28,685, with 111 impressions and 0 clicks in Bing. These protocols solve discovery — helping machines find and parse content. They do not create authority, and authority is what decides whether a search engine bothers to index you.

What is Cloudflare Crawler Hints and should I turn it off?

It automatically submits changed URLs to IndexNow on your behalf. On my site it pushed 209,600 URLs, about 4,900 every 18 hours, including noindex pages and a 404. Bing flagged it as excessive. If your site is large or template-generated, check what it is actually submitting before leaving it on.

Why would my CDN change my robots.txt?

Cloudflare can inject a managed block that disallows AI crawlers, and it appears above your own rules. Mine was blocking GPTBot, ClaudeBot, CCBot and Google-Extended while my own file explicitly welcomed them, and declared ai-train=no against my ai-train=yes. Always read the rendered file at your live URL, not the source on disk.

Is IndexNow worth setting up?

Yes, for genuinely changed content — it is a key file plus one HTTP POST, and Bing, Yandex, Naver and Seznam all consume it. Google does not. Just do not bulk-resubmit an unchanged site; that is the behaviour that gets a key deprioritised.

What should a small site do first?

Fix what is broken, publish llms.txt and a question-answering endpoint, audit what your CDN is doing by default, and then spend your remaining effort earning links. The technical work is a weekend. The links are the years.