Use noindex to request exclusion, robots.txt to manage crawling
Use noindex when a public URL should be crawled but should not appear in Google Search. Use robots.txt when you want to ask compliant crawlers not to fetch a path. They solve different problems, and blocking a URL in robots.txt can prevent a crawler from seeing its noindex directive. Google’s robots documentation describes robots.txt as a crawl-control file; its robots meta documentation says an indexing directive can only be followed when the crawler can access the page.
Start with the outcome. For a temporary campaign page that must stay public to people but absent from results, allow crawling and return noindex in HTML or an HTTP header. For a server endpoint whose responses should not be fetched by cooperative crawlers, use a robots rule and protect it separately if it is private. Neither mechanism is authentication.
Walk through the URL state
Make a short inventory with the URL, intended audience, response status, robots rule, and index directive. Check both https://example.com/ and any www hostname because robots files are host-specific. Fetch the URL as an ordinary client, inspect the final response after redirects, and record whether the directive is in the HTML head or X-Robots-Tag header. Then check the applicable robots group with the robots.txt AI checker.
Suppose Northwind publishes /wholesale-price-list/ for signed-in distributors. The team adds Disallow: /wholesale-price-list/ and assumes it is hidden. The URL still returns a 200 page to anyone, so it is neither access control nor a reliable removal method. The better fix is a login that returns the appropriate protected response. If the page instead needs to remain reachable but unindexed while a redesign is reviewed, remove the disallow rule and send X-Robots-Tag: noindex until the review ends.
Avoid the common crossed wires
Do not combine Disallow and noindex on the same URL expecting the second rule to rescue the first. Do not put credentials, internal report names, or security-sensitive paths in a public robots file. Do not call a request with a familiar User-Agent verified crawler activity; a header can be copied. Review the distinction in robots policy is not enforcement.
Also avoid sitewide noindex as a staging shortcut that is later copied to production. Keep environment configuration explicit and test a representative public URL after every deployment. The AI search readiness checker can surface raw response evidence, but it does not establish a search engine’s final index state.
Preserve the evidence
Keep a dated before-and-after capture of headers and robots content when changing a live URL. Test public pages, exclusion candidates, and authenticated boundaries separately. That makes rollback possible if a shared template changes a whole section. A maintainer should be able to explain the result from the delivered response, rather than from an assumed interaction between directives.
FAQ
Does robots.txt remove a URL already indexed?
No. It controls compliant crawling preferences; use accessible noindex, removal workflows, or a proper status response for the intended result.
Is noindex a security control?
No. It is a crawler instruction. Require authentication for material that must not be public.