noindex vs robots.txt Disallow: Which to Use, and the Mistake That Keeps Pages in Google

🤖 Crawling · Updated October 2026 · 11 min read · by Jacob Campbell

Use noindex when you want a page kept out of search results, and use a robots.txt Disallow when you want crawlers to stop fetching URLs; they are not interchangeable. A disallowed URL can still be indexed and shown in Google without a description, and a page that is both disallowed and tagged noindex often stays in Google, because the crawler is never allowed to fetch the page and see the noindex.

That combination is the most common indexing mistake we see on small business sites, usually on thank-you pages, internal search results and staging copies. This guide explains what each control does according to Google’s documentation and the robots.txt standard (RFC 9309), the difference between the meta tag and the X-Robots-Tag header, how AI crawlers and Google’s AI features respond to each, and a decision table for the usual cases.

Key facts

• Disallow — Controls crawling, not indexing
• noindex — Only works if the crawler can fetch the page
• Sep 1, 2019 — Google stopped honouring noindex in robots.txt
• ~6 months — How long a Search Console temporary removal lasts

The short answer: crawling and indexing are different steps

robots.txt controls whether a crawler may fetch a URL; noindex controls whether a search engine may show a page in its results, and each one only does its own job.

A search engine first discovers a URL (from a link, a sitemap or elsewhere), then crawls it (downloads it), then indexes it (stores and shows it). robots.txt acts at the crawl step. Google’s robots.txt introduction is explicit: it “is not a mechanism for keeping a web page out of Google.” noindex acts at the index step, and Google’s block-indexing guide is equally explicit that it only works when the crawler can read it.

• Want it out of search results? Allow crawling and add noindex.
• Want crawlers to stop spending requests on it? Disallow it in robots.txt, and accept that the bare URL might still appear.
• Want it private? Neither. Put it behind a login or HTTP authentication.

What robots.txt Disallow actually does

A Disallow rule asks compliant crawlers not to request matching URLs; it does not remove those URLs from an index, and Google can still list them if it finds links to them.

Google’s documentation says that if a page is blocked by robots.txt, “its URL can still appear in search results, but the search result won’t have a description.” That is the familiar “No information is available for this page” result. It happens when other pages, on your site or elsewhere, link to the blocked URL: Google knows the address and the anchor text, but not the content.

The robots.txt format is standardized as RFC 9309, the Robots Exclusion Protocol, published by the IETF in September 2022. A few details from it matter in practice:

• The standard is about access by crawlers. It says nothing about indexing, and it states that the protocol “is not a substitute for valid content security measures.”
• When rules conflict, the most specific match (the longest path) wins, and if an Allow and a Disallow match equally, Allow wins.
• If robots.txt returns a 4xx status, crawlers may crawl everything; if it returns a 5xx or cannot be reached, they should assume everything is disallowed.
• Crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, and they must parse at least the first 500 kibibytes.

A typical, sensible robots.txt for a small business site:

User-agent: *
Disallow: /search
Disallow: /cart/
Disallow: /*?sort=
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Note what is not here: thank-you pages, admin URLs and staging folders. Listing a sensitive path in robots.txt also advertises it to anyone who reads the file.

What noindex does, and why it needs the page to be crawlable

noindex tells search engines that support it not to show a page in results, but a crawler can only obey it after fetching the page, so the URL must not be blocked in robots.txt.

Google’s wording leaves no room for doubt: if robots.txt blocks the URL, “the crawler will never see the noindex rule, and the page can still appear in search results.” The robots meta tag reference repeats it: rules on a page disallowed from crawling “will not be found and will therefore be ignored.”

This is the mistake in the headline. Someone wants a page gone, adds noindex, then “to be safe” also disallows it in robots.txt. Google stops fetching the page, never sees the new tag, and keeps the old indexed copy, or a description-less URL. The fix is counter-intuitive but simple:

1. Remove the Disallow rule for that URL.
2. Keep the noindex on the page and make sure the page returns 200 with the tag in the delivered HTML.
3. Request indexing in Search Console’s URL Inspection tool, or wait for a recrawl. Google notes that a revisit can take months for unimportant pages.
4. Once the URL has dropped out (the Page indexing report shows “Excluded by ‘noindex’ tag”), you may add the Disallow back if crawl load is a concern, but there is usually no need.

Two related points. noindex is respected by Google and Bing, but Google notes that other search engines may interpret it differently. And noindex with nofollow is rarely what you want on a normal page; plain noindex still lets crawlers follow its links.

Meta robots tag vs X-Robots-Tag header

The meta robots tag and the X-Robots-Tag HTTP header carry the same rules; the meta tag works only in HTML, while the header works for any file type, including PDFs, images and Markdown.

In HTML, place the tag in the <head>:

<meta name="robots" content="noindex">
<!-- Google only: -->
<meta name="googlebot" content="noindex">

For any response, send a header. In PHP, before any output:

header("X-Robots-Tag: noindex", true);

For every PDF on an Apache site (from Google’s own example):

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

And on nginx:

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

Other rules worth knowing from the same reference: none equals noindex, nofollow; nosnippet blocks text snippets; max-snippet:[number] caps snippet length; max-image-preview:none|standard|large controls image previews; noimageindex keeps images on the page out of Google Images; unavailable_after:[date] drops a page after a date; indexifembedded allows content embedded through iframes to be indexed despite a noindex. The HTML attribute data-nosnippet on a span, div or section keeps just that part out of snippets.

Check headers with curl -sI https://www.example.com/file.pdf | grep -i x-robots, and check the rendered tag with Search Console’s URL Inspection, because a tag added by JavaScript is only seen after rendering.

noindex in robots.txt, and the Removals tool: two things that do not work long term

A Noindex: line in robots.txt has done nothing in Google since September 1, 2019, and the Search Console Removals tool only hides a URL for about six months.

Google announced in July 2019, in a note on unsupported rules in robots.txt, that it was retiring code that handled unsupported and unpublished rules such as noindex, effective September 1, 2019. The alternatives it listed are still the right ones: noindex in the meta tag or HTTP header, a 404 or 410 status, password protection, a Disallow (with the caveats above), and the Removals tool for temporary removal. If you inherit a robots.txt with Noindex: lines, delete them and implement real noindex on the pages.

The Removals tool is for emergencies. A successful request lasts about six months, and Google says blocking a URL there does not stop crawling, only showing. To make removal permanent you must also do one of: delete the content so the URL returns 404 or 410, put it behind a password, or add noindex. Use the tool to hide something urgently, then fix the page properly the same day.

How AI crawlers and AI search features treat each control

For AI crawlers, robots.txt is the main documented control; for Google’s own AI features, the controls are the same noindex and snippet rules that govern Search.

Google AI Overviews and AI Mode. Google’s AI features documentation says a page must be indexed and eligible to be shown with a snippet to appear as a supporting link, and that you limit use with nosnippet, data-nosnippet, max-snippet or noindex. The robots meta tag reference adds that nosnippet also prevents content from being used as a direct input for AI Overviews and AI Mode. So noindex removes a page from both, and nosnippet keeps it in classic results while keeping its text out of the AI answers (at the cost of a plain result without a snippet).

Google-Extended. This robots.txt product token controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini apps. Google says it does not affect inclusion or ranking in Google Search, and it has no separate user agent; it is a robots.txt token only. Disallowing Google-Extended does not remove you from AI Overviews; that is controlled by the Search rules above.

Other AI crawlers. OpenAI, Anthropic, Perplexity and others publish user agent names and say their crawlers respect robots.txt, typically with separate agents for training, search indexing and user-triggered fetches. Two caveats. OpenAI’s documentation, revised in December 2025, says of ChatGPT-User that because its fetches are initiated by a user, robots.txt rules may not apply. And most AI providers do not document whether they honour the meta noindex tag, so do not rely on it to keep content away from AI systems. Bing, which supplies results to several assistants, announced in September 2023 that noarchive keeps content out of Bing Chat (now Copilot) answers and out of training of Microsoft’s generative models, and that nocache limits use to URL, title and snippet.

A robots.txt group that blocks AI training while keeping search features looks like this:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

The full list of agents and what each one does is in our AI crawler user-agent list, and the trade-offs are in robots.txt for AI crawlers. robots.txt is a request, not enforcement; if you need enforcement, see Cloudflare AI Crawl Control.

Staging sites, parameters and other common cases

Staging sites need authentication, parameter URLs usually need a canonical or a Disallow, and pages you want out of results need noindex on a crawlable URL.

Staging and development sites. Neither robots.txt nor noindex is the right tool. A Disallow still lets the URLs be indexed if anyone links to them, and a noindex that is copied to production by accident deindexes the live site, which is a classic launch-day disaster. Use HTTP authentication so crawlers get a 401:

# .htaccess on the staging host
AuthType Basic
AuthName "Staging"
AuthUserFile /home/account/.htpasswd
Require valid-user

As an extra layer, send X-Robots-Tag: noindex from the staging server config (not from the code base that gets deployed), so the header cannot follow the code to production.

URL parameters. Tracking parameters (?utm_source=) should stay crawlable with a canonical to the clean URL; see canonical tags explained. Sort and filter parameters that multiply into thousands of URLs are a crawl problem, and Google’s faceted navigation guidance recommends robots.txt for those. Internal site search results are best disallowed, because they can generate endless URLs.

Thank-you and confirmation pages. noindex, crawlable. Old pages with no replacement. Return 410 or 404 rather than noindex; a removed page is cleaner than a hidden one. Old pages with a replacement. A 301 redirect. Login and account pages. Require authentication; add noindex to the public login form if you do not want it listed.

Decision table: which control to use

Choose the control by the outcome you want: out of results, less crawling, private, or gone.

• Keep a public page out of Google results (thank-you page, tag archive, thin page): noindex meta tag, page crawlable.
• Keep a PDF, image or Markdown file out of results: X-Robots-Tag: noindex header, file crawlable.
• Stop crawlers wasting requests on internal search, endless filters or calendars: robots.txt Disallow; the URLs may still be listed without descriptions.
• Remove a page that already shows in Google: noindex or 404/410, with no Disallow until it has dropped out; add the Removals tool if it is urgent.
• Keep staging or client areas private: HTTP authentication or login. Not robots.txt.
• Consolidate duplicates (parameters, print views, http/https): 301 redirect or rel="canonical", not noindex.
• Stay in Google but keep text out of AI Overviews and AI Mode: nosnippet, data-nosnippet or max-snippet.
• Opt out of Gemini training and grounding without affecting Search: Disallow for Google-Extended.
• Opt out of other AI training crawlers: robots.txt groups for their training user agents, with enforcement at the CDN if you need certainty.

Limits. Every control here is voluntary for the crawler except authentication. Changes take effect only when the URL is recrawled. And both controls are blunt: a noindex on the wrong template or a Disallow: / left over from development can remove a whole site, so check robots.txt and a sample of templates after every deploy. Our why isn’t my website ranking guide starts with exactly those checks.

Product facts: the AI Indexing Pack

The AI Indexing Pack includes bot access control, which is the enforcement layer robots.txt itself cannot provide. Stated plainly, so it can be checked against the product page:

• What it is: the AI Indexing Pack is a set of files you upload to your own website. It is sold by Built2WinWeb as a one-time purchase, with no subscription.
• What it ships: (1) a working MCP server using the Streamable HTTP transport, with typed output schemas and tool annotations; (2) Markdown content negotiation, so agents can request a clean text version of each page; (3) llms.txt, agents.txt and agent cards; (4) the full /.well-known/ discovery tree; (5) OAuth protected-resource metadata; (6) bot access control that verifies crawler identity; (7) a README with a verification checklist.
• Requirements: PHP 7.4 or newer, Apache or LiteSpeed with .htaccess support, FTP access, and a domain you control. Installation takes about 30 minutes.
• What it is not: it is not a WordPress plugin (that is the separate WordPress edition, which needs WordPress 6.0+ and PHP 8.0+), it does not write content for you, it does not guarantee rankings or citations, and it is not a managed service.
• Free routes in: sites built on AIWebBuilding get the core built in, and a 50-day streak on FreeAIBoost unlocks the pack for one domain. Current prices are on the product pages.

Related reading

• robots.txt for AI crawlers — Which AI bots to allow, which to block, and the trade-offs.
• AI crawler user-agent list — Every AI bot, who runs it and what it does.
• Canonical tags explained — How to fix duplicate content with redirects and rel=canonical.
• Cloudflare AI Crawl Control — Enforcing bot rules at the edge instead of asking politely.
• Show up in AI Overviews and AI Mode — What Google actually says about eligibility and controls.
• Why isn’t my website ranking on Google? — Nine real reasons, starting with the indexing checks.
• AI Indexing Pack — The file-drop kit: MCP server, Markdown pages, llms.txt, discovery files and bot access control for any PHP site.

Frequently asked questions

Does robots.txt Disallow remove a page from Google?

No. Disallow only stops crawling. Google can still index a disallowed URL it finds through links and show it in results without a description.

Why is my noindex page still in Google?

The most common reason is that the URL is also disallowed in robots.txt, so Googlebot never fetches the page and never sees the noindex. Remove the Disallow, keep the noindex, and let Google recrawl the page.

Does noindex in robots.txt still work?

Not in Google. Google stopped supporting noindex rules in robots.txt on September 1, 2019; use a meta robots tag or an X-Robots-Tag HTTP header instead.

How do I noindex a PDF?

Send the HTTP header X-Robots-Tag: noindex with the PDF, for example with an Apache Files block or an nginx location block. The file must stay crawlable so the header can be read.

How do I keep my content out of Google AI Overviews but stay in search?

Use nosnippet, data-nosnippet or max-snippet. Google says nosnippet also prevents content being used as a direct input for AI Overviews and AI Mode, while noindex removes the page from Search entirely.