Let verified AI crawlers in without opening the door to scrapers

For builders managing AI discovery, the hard part is not adding another file to the web root. It is deciding which automated requests should reach the site, which public material they may fetch, and how to keep the rest of the application off limits.

That makes bot access control a practical infrastructure task, not a promise that a site will appear in an AI answer. A crawler needs to discover and fetch public content. A scraper may send similar requests while ignoring site preferences. The request itself does not tell you what the client will do with the material.

Start with the limit of a bot name

A User-Agent string can help classify traffic, but it is just text supplied with a request. A client can copy a familiar crawler’s name. A rule that allows every request claiming to be an AI search bot can therefore admit unauthorized scrapers as well.

Use a bot name as an initial signal, not proof of identity. Where a crawler operator publishes stable IP ranges or verification instructions, compare requests against that documentation. Reverse DNS checks can add evidence when the operator supports them; confirm the result with a forward lookup rather than trusting a reverse name alone. These checks take maintenance, and not every agent or crawler publishes a reliable range.

Keep the policy narrow when identity is uncertain. A new or unverified crawler does not need access to account pages, admin routes, private files, or expensive application actions to read public business information.

Separate discovery from access control

Robots.txt can express crawl preferences to cooperative bots. llms.txt can help describe a site’s material for agent discovery. Neither is an access-control boundary. A bot can ignore either file and request a URL directly. Server rules, application authorization, and network controls decide whether that request is served.

Response headers have a related but different job. Directives such as X-Robots-Tag can communicate indexing preferences in a response; they do not stop a client from making the request in the first place. Do not treat a header, a manifest, or a crawler’s claimed identity as a substitute for access checks.

For Apache sites, teams often search for ways to block AI scrapers with .htaccess. A copied rule can be a poor shortcut: matching on a broad User-Agent substring may block benign traffic and still miss a scraper that changes its name. Rules also depend on the site’s Apache configuration and the order in which access decisions are applied. Review the intended match, test it against real requests, and keep a rollback path.

Give public content a measured route

Build the policy around resources, not labels alone. Make the pages and discovery endpoints intended for public reading reachable. Keep authentication, customer data, internal search, and write-capable endpoints behind their existing authorization checks. If a crawler only needs rendered public pages, do not give it a broader route into the application.

Then set practical limits. Rate limits, request-size caps, caching, and restrictions on costly query patterns can reduce unnecessary origin work. Apply them with care: a limit that is too aggressive can disrupt legitimate fetches, while a generous limit can leave the server exposed to bursts. Watch request volume, response codes, latency, and server errors before changing the thresholds.

Test from outside the trusted network. Check the public pages and agent discovery routes that should work, then check a private route and an unwanted path that should fail. Repeat through the CDN, firewall, or proxy in front of the origin; a rule at one layer can be overridden or contradicted by another. Keep logs long enough to see whether a rule is blocking the traffic it was meant to block.

Keep discovery and security in the same review

Allowing an AI crawler is not the same as making every endpoint public. Nor does blocking a suspicious client require hiding the whole site from agents. A useful policy grants access to selected public material, denies protected routes regardless of bot identity, and treats unverified clients conservatively.

Teams working on WordPress can use the earlier guidance on testing AI discovery routes against firewall and cache rules as a route-level checklist. The same principle applies beyond WordPress: verify the path through every layer that can serve or deny a request.

BuiltToWinWeb describes WebAgentScan as a hosted service that scores website readiness for AI agents out of 100. A readiness score can help teams track an assessment, but it should not replace direct access-control tests. Confirm what an unauthenticated request can actually retrieve, and whether the server stays healthy under the traffic the policy permits.

The builder’s monthly check is straightforward: review crawler identity evidence, inspect the allow and deny rules, test the public discovery routes, and compare logs and load with the prior period. If the evidence does not support trusting a crawler, keep the boundary in place. Discovery should be deliberate, not an all-or-nothing trade between visibility and security.