# ============================================================
# RSL - Really Simple Licensing 1.0
# Global directive (not scoped to any user-agent group). Crawlers must
# retrieve and comply with these terms before processing site content.
# Spec: https://rslstandard.org/rsl
# ============================================================
License: https://built2winweb.com/license.xml

# ============================================================
# CONTENT SIGNALS (Cloudflare Content Signals Policy)
# search   = appear in a traditional search index (links + excerpts)
# ai-input = use content in real-time generative answers (RAG)
# ai-train = use content to train or fine-tune a model
#
# All three are YES. This site WANTS to be read, cited and surfaced by
# AI assistants - that is the entire point of the agent surface it
# publishes (MCP, A2A, NLWeb, llms.txt). Signals are declarative, not
# enforcement. The RSL License directive above carries the legal terms.
# ============================================================

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

# ------------------------------------------------------------
# LANGUAGE VARIANTS (?lang=xx) — NOT CRAWLED.  Added 2026-09-13.
#
# WHY THIS IS HERE AND WHY IT IS NOT THE MISTAKE DOCUMENTED FURTHER DOWN
# Read the "Pruned URL families" note below before touching this. That note
# says, correctly, not to robots-block a URL that carries a meta tag Google has
# to FETCH in order to obey. The ?lang= variants are the opposite case, on
# every point that mattered there:
#
#   - They carry no noindex. They serve <meta robots="index, follow"> and a
#     rel=canonical pointing at the bare English URL. Nothing is being asked of
#     Googlebot that a block would prevent it from hearing.
#   - The consolidation has ALREADY happened. Search Console reports 6,583 of
#     them as "Alternate page with proper canonical tag" -- Google has read the
#     canonical, accepted it, and excluded them. There is no deindexing left to
#     do that this block could defeat.
#   - Their "follow" value is nil. The outgoing links on a ?lang= page are
#     either the same ten header destinations every English page already links,
#     or more ?lang= URLs. No link equity reaches anywhere it was not already
#     going.
#
# WHAT IT COSTS AND WHAT IT BUYS
# The sitemap carries 28,779 URLs. Every one of them is reachable in eleven
# languages through the switcher, so the crawlable surface was about 316,000
# URLs, of which roughly 287,000 can never rank because they canonicalise away.
# Meanwhile Cloudflare rate limiting answers 429 after ~70 requests in ~10s per
# IP (measured 2026-09-13: the 71st sequential request tripped it, Retry-After
# 10). Googlebot cannot crawl a 316,000-URL site through a 70-per-10s gate, and
# Search Console shows the result -- 679 "Server error (5xx)", 36,953 "Crawled
# - currently not indexed", 14,125 "Discovered - currently not indexed", and
# breadcrumb-eligible items down from ~9,000 in June to 57.
#
# This is exactly the lever the pruned-families note names: "reduce the NUMBER
# of these variants". It reduces them to zero for crawlers while leaving every
# translation working for the people who click the switcher.
#
# WHY TWO PATTERNS, AND WHY NOT "&lang=".
# "/*?lang=" catches the ordinary case, where lang is the first parameter.
# The obvious companion, "/*&lang=", DOES NOT WORK -- verified against a
# Google-spec parser (protego), which allows /a/?utm=1&lang=es straight
# through it. "/*?*lang=" is the one that matches a lang parameter in any
# position, and it still requires a query string, so it cannot accidentally
# catch a path segment that happens to contain "lang=". Do not "simplify"
# this to a single "&" rule; it silently stops blocking anything with a
# tracking parameter in front of the language.
# Under RFC 9309 the longest match wins, and both are longer than "Allow: /".
#
# TO REVERT: delete the two Disallow lines, and drop rel="nofollow" from the
# switcher anchors in includes/site-header.php, index.php, web-design.php,
# web-design-city.php, article-layout.php and state-of-ai-agent-readiness.php.
# ------------------------------------------------------------
Disallow: /*?lang=
Disallow: /*?*lang=

# ------------------------------------------------------------
# Machine-readable discovery documents (added 2026-08-08)
# CRITICAL: these MUST out-rank the "Disallow: /*.json$" rule further
# down. Under RFC 9309 the longest matching pattern wins, and every
# path here is longer than the 8-character "/*.json$", so these win.
# Without them robots.txt was forbidding every manifest the site
# publishes - agents.json, openapi.json, server.json and the entire
# .well-known tree (agent-card, acp, mcp, agent-skills, permissions).
# A perfect manifest on a disallowed path is an invisible manifest.
# ------------------------------------------------------------
Allow: /.well-known/
Allow: /agents.json$
Allow: /agents.txt$
Allow: /openapi.json$
Allow: /server.json$
Allow: /license.xml$
Allow: /llms.txt$
Allow: /llms-full.txt$
Allow: /AGENTS.md$
Allow: /sitemap.md$

# Editing backups (*.bak-*). .htaccess already 403s these, so every fetch is a
# wasted round trip. The .well-known variant is spelled out in full because the
# blanket "Allow: /.well-known/" above is a longer match and would win otherwise.
Disallow: /*.bak*
Disallow: /.well-known/*.bak*

# Allow essential assets
Allow: /fa/css/
Allow: /fa/webfonts/
Allow: /js/
Allow: /*.webp$
Allow: /*.css$
Allow: /*.js$

# Allow IndexNow key file (must be publicly crawlable for verification)
Allow: /b2w9a4f3c7e1d8b6e5f2a0c3d7e9f1b4.txt

# Disallow cache — hundreds of JSON files waste crawl budget
Disallow: /cache/

# Disallow sensitive and non-content directories
Disallow: /state_blocks/
Disallow: /tmp/
Disallow: /logs/
Disallow: /admin/
Disallow: /PHPMailer/
Disallow: /PHPMailer-6.8.0/
Disallow: /stripe-php-14.10.0/
Disallow: /vendor/
Disallow: /composer/

# Disallow non-public utility scripts
Disallow: /generate_cities.py
Disallow: /generate_city_details.php
Disallow: /split-city-details.php
Disallow: /test_place_id.php
Disallow: /test_weather.php
Disallow: /debug.php
Disallow: /ctr-graph.php
Disallow: /InvoiceSystem.php
Disallow: /send_invoice.php
Disallow: /create_stripe_session.php
Disallow: /stripe_success.php
Disallow: /send_contact_email.php
Disallow: /notify-indexnow.php

# Disallow raw data files
Disallow: /*.csv$
Disallow: /*.json$
Disallow: /*.zip$

# ------------------------------------------------------------
# Decorative trust badges - crawl-budget sink (added 2026-08-01)
# Cloudflare AI Crawl Control showed Claude-SearchBot pulling these
# files ~1,350x / 3.7 MB in 24h, re-fetching despite
# Cache-Control: max-age=31536000, immutable. They already carry
# X-Robots-Tag: noindex,nofollow via .htaccess, so they have zero
# index value and nothing is lost by blocking the fetch.
# Longer path match beats the "Allow: /*.webp$" rule above.
# ------------------------------------------------------------
Disallow: /goodfirms.webp
Disallow: /yelp.webp
Disallow: /sitelock.webp
Disallow: /trustpilot.webp
Disallow: /norton.webp
Disallow: /bbb.webp
Disallow: /google.webp
Disallow: /upwork.webp
Disallow: /hostinger.webp
Disallow: /pcs.webp
Disallow: /blank.gif

# ------------------------------------------------------------
# Pruned URL families - state/city variants
#
# The 31 "Disallow: /<family>/*/" lines that used to sit here were REMOVED on
# 2026-09-06, deliberately. Please read this before putting them back.
#
# Those same 31 families are served <meta robots="noindex, follow"> by
# config/pruned-slugs.php. Blocking them here meant Googlebot never fetched
# them, and a page that is never fetched is a page whose meta tag is never
# read. The block defeated BOTH halves of that tag:
#
#   noindex - never seen, so the URLs could still be indexed URL-only from any
#             external link. The block did not reliably deindex anything.
#   follow  - never seen, so no link equity moved through these pages to the
#             national page each one points at. That flow is the whole reason
#             the tag says "follow", and it was silently doing nothing.
#
# The crawl-budget saving was real (Cloudflare AI Crawl Control measured 0
# referrals across 4,798 of these paths) and giving it up is the cost of this
# change. It buys reliable deindexing and restores internal authority to the
# national pages, which is where the rankings actually are.
#
# If crawl budget becomes the bigger problem again, the lever is to reduce the
# NUMBER of these variants, not to block URLs carrying a meta tag Google has to
# fetch in order to obey.
# ------------------------------------------------------------
# ============================================================
# AI / LLM CRAWLERS — explicitly welcomed so the full site can be
# read, cited, and surfaced by AI search engines and assistants.
# Same protections as the default group (sensitive dirs stay blocked).
# ============================================================
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: YouBot
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
# ------------------------------------------------------------
# LANGUAGE VARIANTS (?lang=xx) — NOT CRAWLED here too. Added 2026-09-13.
# Full reasoning is on the identical pair in the "User-agent: *" group above;
# it is not repeated here. The short version: every ?lang= URL canonicalises to
# the bare English URL, so an AI crawler that follows one is reading a duplicate
# of a page it can already reach, and eleven of them per page is what put the
# crawlable surface at ~316,000 URLs against a 429 gate. Blocking them costs no
# content coverage -- the same pages remain fully open in English.
# ------------------------------------------------------------
Disallow: /*?lang=
Disallow: /*?*lang=

# ------------------------------------------------------------
# Machine-readable discovery documents (added 2026-08-08)
# CRITICAL: these MUST out-rank the "Disallow: /*.json$" rule further
# down. Under RFC 9309 the longest matching pattern wins, and every
# path here is longer than the 8-character "/*.json$", so these win.
# Without them robots.txt was forbidding every manifest the site
# publishes - agents.json, openapi.json, server.json and the entire
# .well-known tree (agent-card, acp, mcp, agent-skills, permissions).
# A perfect manifest on a disallowed path is an invisible manifest.
# ------------------------------------------------------------
Allow: /.well-known/
Allow: /agents.json$
Allow: /agents.txt$
Allow: /openapi.json$
Allow: /server.json$
Allow: /license.xml$
Allow: /llms.txt$
Allow: /llms-full.txt$
Allow: /AGENTS.md$
Allow: /sitemap.md$

# Editing backups (*.bak-*). .htaccess already 403s these, so every fetch is a
# wasted round trip. The .well-known variant is spelled out in full because the
# blanket "Allow: /.well-known/" above is a longer match and would win otherwise.
Disallow: /*.bak*
Disallow: /.well-known/*.bak*

Disallow: /cache/
Disallow: /state_blocks/
Disallow: /tmp/
Disallow: /logs/
Disallow: /admin/
Disallow: /vendor/
Disallow: /composer/
Disallow: /PHPMailer/
Disallow: /PHPMailer-6.8.0/
Disallow: /stripe-php-14.10.0/
Disallow: /*.csv$
Disallow: /*.json$
Disallow: /*.zip$
Disallow: /goodfirms.webp
Disallow: /yelp.webp
Disallow: /sitelock.webp
Disallow: /trustpilot.webp
Disallow: /norton.webp
Disallow: /bbb.webp
Disallow: /google.webp
Disallow: /upwork.webp
Disallow: /hostinger.webp
Disallow: /pcs.webp
Disallow: /blank.gif

# ------------------------------------------------------------
# Pruned URL families - state/city variants
#
# The 31 "Disallow: /<family>/*/" lines that used to sit here were REMOVED on
# 2026-09-06, deliberately. Please read this before putting them back.
#
# Those same 31 families are served <meta robots="noindex, follow"> by
# config/pruned-slugs.php. Blocking them here meant Googlebot never fetched
# them, and a page that is never fetched is a page whose meta tag is never
# read. The block defeated BOTH halves of that tag:
#
#   noindex - never seen, so the URLs could still be indexed URL-only from any
#             external link. The block did not reliably deindex anything.
#   follow  - never seen, so no link equity moved through these pages to the
#             national page each one points at. That flow is the whole reason
#             the tag says "follow", and it was silently doing nothing.
#
# The crawl-budget saving was real (Cloudflare AI Crawl Control measured 0
# referrals across 4,798 of these paths) and giving it up is the cost of this
# change. It buys reliable deindexing and restores internal authority to the
# national pages, which is where the rankings actually are.
#
# If crawl budget becomes the bigger problem again, the lever is to reduce the
# NUMBER of these variants, not to block URLs carrying a meta tag Google has to
# fetch in order to obey.
# ------------------------------------------------------------
# AI guide files (for LLMs / AI assistants — not a robots directive, informational)
# LLM-Content: https://built2winweb.com/llms.txt
# LLM-Content: https://built2winweb.com/llms-full.txt

# Sitemap
# ARD (Agentic Resource Discovery): points agents and readiness scanners
# straight at the capability manifest instead of guessing the path.
Agentmap: https://built2winweb.com/.well-known/ai-catalog.json

Sitemap: https://built2winweb.com/sitemap.xml
Sitemap: https://built2winweb.com/news/sitemap.xml
# The AI website builder lives on its own host and carries its own sitemap.
# Cross-host Sitemap lines are allowed in robots.txt (sitemaps.org, "Sitemaps & Cross Submits").
Sitemap: https://aiwebbuilding.built2winweb.com/sitemap.xml