A practical digest for MCP security and AI crawler limits

For builders working on AI agent discovery, the useful update is a shift in review discipline: treat endpoint access, crawler load and readiness scoring as separate engineering questions. A site can be easy for an agent to discover and still expose too much. It can also have a score that looks strong while leaving important operational risks untested.

There is no basis here for naming a new universal MCP authorization rule or a standard crawler request limit. That matters. Teams should avoid presenting an arbitrary threshold as an industry benchmark. Instead, set and document controls that match the data and operations each site actually exposes.

Make public MCP access a deliberate choice

A reachable endpoint is not automatically a safe endpoint. Start by deciding what an unauthenticated visitor or agent is allowed to do. Public documentation and read-only content carry a different risk from operations that reveal private records or change application state. Keep the exposed surface narrow, and require stronger controls where the impact of a request is higher.

Review authorization at the operation and data level, not just at the server boundary. Check that a request cannot use an allowed route to retrieve records outside its intended scope. For tools that can cause changes, consider whether the action needs confirmation or a separate authorization path. Log enough to investigate unexpected use, while avoiding unnecessary collection of sensitive data.

These are design checks, not a claim that one mechanism fits every MCP implementation. Teams adding discovery routes to existing applications can also use the safeguards in this guide to exposing selected content without leaking private records.

Measure crawler limits before choosing numbers

A rate limit is useful only if it reflects real traffic and protects a real resource. A single cap for every path can penalize normal page discovery while failing to protect an expensive search or data endpoint. Measure requests by route and over time. Track bursts, concurrency, response latency and error rates alongside total request volume.

Set separate policies where the cost or sensitivity differs. Make the limit observable to operators, and decide what a client should receive when it exceeds the allowance. Revisit the policy after traffic changes rather than treating the first configured number as permanent. If clients cannot be reliably identified, do not assume a user-agent string alone provides strong identity; use it as one signal among others.

There is no defensible numeric limit to prescribe for every site. A small publisher, a product catalog and a service exposing structured data have different workloads. The practical benchmark is whether the team can explain why its limit exists, see when it is being reached and tell legitimate demand from harmful or wasteful traffic.

Read readiness scores as evidence, not clearance

WebAgentScan is described as scoring a domain out of 100 across seven graded areas, including AI agent readiness, MCP server security and Domain Rating. That makes a score a useful summary for comparison and follow-up. It does not establish, by itself, that an endpoint's authorization model is sound or that its rate limits will hold under load.

Ask what sits behind any aggregate score. Which checks passed? Which failed? What did the scan actually observe, and what needs an authenticated or operational test? A site can improve discoverability while still needing a separate review of permissions, data boundaries and abuse handling. Keep those findings distinct in an audit instead of allowing one high number to stand in for all of them.

For teams that use readiness data alongside search evidence, a combined audit can show where visibility and agent access diverge. That distinction is useful because rankings, crawl access and endpoint safety answer different questions.

A builder's monthly checklist

• Inventory each public MCP operation and the data it can return or change.
• Test authorization boundaries with requests outside the intended scope.
• Measure request volume and resource cost by route before setting crawler limits.
• Record what clients and operators see when a limit is reached.
• Keep scan scores, security tests and live traffic monitoring as separate evidence.

The category's durable lesson is simple: agent discovery should make useful content reachable, not make every interface unrestricted. A score can point to work. A policy can limit load. Only a deliberate authorization design determines what an agent is allowed to access.