Content Signals and RSL Explained: Machine-Readable Rules for Search, AI Answers and AI Training

📜 Licensing · Updated October 2026 · 4 min read · by Jacob Campbell

robots.txt answers one question: may this crawler fetch this path? It says nothing about what happens to the content afterwards, whether it is shown in a search result, quoted in an AI answer, or used to train a model. Two newer mechanisms fill that gap: Content Signals, a robots.txt extension for stating preferences on search, AI input and AI training, and RSL (Really Simple Licensing), a standard for publishing machine-readable licence terms. This guide explains both, gives examples, and is straight about what they can and cannot enforce.

Key facts

• 3 signals — search, ai-input and ai-train
• 2025 — Content Signals Policy and RSL both announced
• Preference — Signals state intent; they are not technical enforcement
• Licence — RSL adds terms, including paid ones

Content Signals

Cloudflare introduced the Content Signals Policy in September 2025 and began adding it to the robots.txt files it manages for customers. It adds a Content-Signal line to robots.txt with three keys:

• search: building a search index and showing links and short excerpts.
• ai-input: using content as input to an AI answer in real time (retrieval, grounding).
• ai-train: training or fine-tuning models.

Each is yes or no; a missing key expresses no preference. Example:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

This site publishes search=yes, ai-input=yes, ai-train=yes, because for a web agency being known to models is distribution.

Why the split matters

The three uses have very different value for a business. Search and AI input send visitors and citations; training benefits you only indirectly and later. Content Signals lets you say “answer questions with my content, but do not train on it”, which robots.txt alone cannot express because several AI crawlers serve more than one purpose.

RSL: Really Simple Licensing

RSL was announced in September 2025 by the RSL Collective, whose founders include a co-creator of RSS, with support announced by a number of large publishers and platforms. It defines a machine-readable licence document that states what AI systems may do with content and on what terms: free with attribution, by subscription, pay per crawl, or pay per inference. A site points to its licence from robots.txt or a Link header, and can attach licences to feeds and individual pages.

We publish an RSL licence on this site as an experiment; it is listed on our agent-ready page.

The honest limits

• Neither is enforcement. Like robots.txt, they state terms; compliance depends on the crawler operator. Some operators have said they will respect signals; others have not committed.
• They do not reach backwards. Content already collected is not affected.
• Enforcement lives elsewhere: CDN bot management, verified-bot programmes and pay-per-crawl schemes are the technical side. See robots.txt for AI crawlers for verification.

What to publish

1. Decide the three answers. Most businesses want search=yes, ai-input=yes; ai-train is a values call.
2. Add the Content-Signal line to robots.txt and check your CDN is not overwriting it with a managed file.
3. Make robots.txt user-agent rules consistent with the signals (do not say ai-input=yes and then block every AI search bot).
4. If you publish valuable original content, consider an RSL licence.
5. Check your logs to see who respects what: AI crawlers in server logs.

Related reading

• robots.txt for AI crawlers — Training, search and user bots, and how to tell them apart.
• Find AI crawlers in your server logs — Which bots visit, what they fetch and what they get back.
• Agent discovery files explained — agents.txt, agent cards, server cards and the .well-known tree.
• How this site is agent-ready — Our own live setup: the MCP endpoint, llms-full.txt, Markdown twins and the .well-known tree.
• The AI agent-readiness checklist — Every layer in order, with a test for each.
• How to get cited by AI assistants — What actually makes ChatGPT, Claude and Perplexity name you.
• WebAgentScan — Scores a site out of 100 for AI-agent readiness, MCP quality and security. Paid plans from $9 a month.

Frequently asked questions

What are Content Signals?

A robots.txt extension introduced by Cloudflare in September 2025 that lets a site state yes or no preferences for search, ai-input and ai-train uses of its content.

What is the difference between ai-input and ai-train?

ai-input covers using content to answer questions in real time; ai-train covers training or fine-tuning models.

What is RSL?

Really Simple Licensing, a standard announced in September 2025 for publishing machine-readable licence terms that tell AI systems how content may be used, including paid terms.

Are Content Signals legally or technically enforced?

No. They state preferences. Compliance depends on crawler operators; technical enforcement needs bot management or verification.

Can I allow AI answers but block training?

Yes. Content-Signal: search=yes, ai-input=yes, ai-train=no expresses exactly that, ideally with matching robots.txt rules for training crawlers.