Keep WordPress AI discovery routes open without loosening security

Adding agent discovery files to a WordPress site is only half the job. The routes also need to remain reachable after security plugins and server caching are doing theirs. The safe approach is to install the WordPress AI Indexing Pack, record what it publishes, and test those resources from outside the site before changing any blocking rule.

The pack is described as a plugin that generates discovery files from the site’s own pages, posts and settings. It includes an MCP server and a checker that tests whether endpoints are live from outside. It does not touch theme files or .htaccess. That reduces some installation risks, but it does not guarantee that a firewall, host-level rule or cache will treat every route as intended.

Start with a route inventory

Install and activate the plugin according to its instructions. Before adjusting Wordfence, iThemes or server settings, use the plugin’s documentation and checker to identify the exact URLs it has published. The pack describes files such as llms.txt, agents.txt, agent cards and .well-known resources, as well as an MCP server. Do not guess an MCP path or assume every site uses the same URL.

Record each URL and the expected response. A plain text discovery file, a Markdown response and an MCP endpoint may not behave alike. That distinction matters when a firewall rule or cache policy is being reviewed. Static manifests describe a site; an MCP endpoint provides a different kind of interface. The paper’s earlier guide on MCP security and crawler limits covers why public interfaces need explicit boundaries.

Check security rules before changing them

With the routes identified, test them from a network outside the WordPress installation. Start with the plugin’s external checker, then request the listed URLs directly in a browser or with a standard HTTP client. Note the response status and whether the returned content matches the intended resource. A successful page load from an administrator’s session is not enough: logged-in access can hide a public-access problem.

If a request fails, inspect the security plugin’s events or logs around that test. In Wordfence or iThemes, look for a block tied to the specific request, rule or rate limit. Do not switch off the firewall to make a test pass, and do not broadly allow all automated clients. If the logs identify a false positive, use the narrowest supported exception for the verified route or rule, then repeat the outside test. Keep a note of the change so it can be reviewed after plugin or rule updates.

Do not treat every denied request as a defect. A security layer may correctly reject malformed traffic or requests that exceed a limit. The goal is to make the intended public resources available while retaining controls on other paths and traffic. If the logs do not explain a failure, ask the host or security-plugin support team before adding a broad bypass.

Review caching separately

A server cache can return an old response, serve the wrong representation or interfere with a request that should reach the application. Test each route both before and after the cache is active, where the hosting setup allows it. Compare status and content, and check whether a fresh request reflects a recent change to the site.

Do not exclude every discovery URL from caching by default. Some generated files may be safe to cache; an application endpoint may need different handling. If testing shows that a cached response is stale or unsuitable, adjust caching only for the affected URL or response type, using the host’s documented controls. Then repeat the same outside checks. Avoid rules based on a guessed path or a blanket exclusion for all .well-known resources.

Make verification repeatable

After each security or cache change, run the plugin’s external checker again and test the routes directly. Save the URL list, expected responses and results in the site’s deployment notes. Recheck after plugin updates, security-rule changes, host migrations and significant content changes. If a route stops working, compare the failure with the last known-good check before changing access controls.

A readiness score from a scanning service can provide another signal about a site’s agent accessibility, but it is not a security audit. The useful operational test is narrower: the intended discovery resources are available from outside, the returned content is correct, and access to unrelated parts of the site remains protected. That is a better standard than making every bot request succeed.