Return the status that describes the resource, not the template
HTTP status codes let a crawler and a browser distinguish an available page from a redirect, missing URL, login requirement, or temporary failure. Google says it uses HTTP status codes to understand crawl problems and moves in its JavaScript SEO documentation; consult its HTTP and network errors guidance for crawler-specific behavior. A polished application shell cannot turn a 200 error page into a reliable missing-page signal.
Audit route classes rather than random URLs: normal content, moved content, permanently retired content, typo routes, protected dashboards, rate-limited API endpoints, and maintenance pages. Capture the initial status, every redirect, final status, response body, robots headers, and cache behavior. Include desktop and mobile hostnames if their routing differs. Status is one layer; canonical tags and links must agree with it.
Apply the pattern to a removal
An online course deletes /courses/legacy-analytics/ and has no comparable replacement. Returning the course template with “This course is gone” and status 200 creates ambiguity. Return 404 or a suitable gone response according to the product’s lifecycle, offer helpful navigation, remove the URL from sitemap and internal links, and avoid redirecting it to an unrelated home page. If the course moved to a close replacement, send one direct permanent redirect and make the replacement page a coherent destination.
Protected pages are different: return the appropriate authentication or authorization response instead of relying on Disallow. Temporary overload or maintenance should not be coded as permanent removal. Use status codes at the server or edge that actually generates the response; a client-side message often arrives after the crawler has already read 200.
Test full responses
Request representative URLs directly and with redirect following. Inspect headers after a CDN, framework, and origin each apply their rules. Pair findings with soft 404 pages and indexing and redirect chains and crawler access. The AI search readiness checker reports a bounded raw-fetch observation, not crawl history or final index state.
Do not return 200 for a generic error page, use redirects to mask server failures, or treat robots.txt as access control. Keep a small status test list in release checks so routing changes cannot quietly flatten all outcomes into the same page.
Maintain a compact status fixture list
Keep one example each for available, moved, missing, protected, and temporarily unavailable routes in release verification. Inspect them after CDN, framework, or origin changes. This inexpensive check catches the common failure where a fallback application shell turns every outcome into a 200 response before anyone notices.
FAQ
Keep status behavior in route-level tests where possible. A visual snapshot cannot reveal whether the server accidentally returned success for a missing page, protected route, or internal failure.
Record expected statuses in deployment notes.
Is a 404 bad for every URL?
No. It is the correct response for a page that does not exist and has no useful replacement.
Can a 200 page contain an error message?
It can, but missing-content routes should return a meaningful error status as well.