Maintain llms.txt as a curated documentation resource: check that its summary is accurate, its links work and its selected documents match the current product. Assign it an owner and update it with the documentation release that changes those facts. Publishing a file once and forgetting it creates another source of stale instructions.
Establish why the file exists
Write down the intended reader and task. For a software library, that might be helping an agent find installation instructions and the current API reference. For a small service business with a few clear public pages, maintaining a separate index may not solve an observed problem.
The llms.txt proposal describes a Markdown resource that provides context and links, including scoped files under paths in its current version. It is a proposal, not a universal crawler requirement. Google’s AI optimization guidance does not require a special AI text file for its generative Search features.
Our recommendation is to keep the file only when you can maintain a useful purpose for it. Do not assign it responsibility for access control, search indexation or citation measurement.
Review the selected information
Check the project name, short summary, important limitations and link descriptions against the actual documentation. Prefer a concise set of authoritative destinations over a dump of every URL on the domain. A reader should understand which version is current and where to find older material when it is still supported.
For each destination, inspect the final response and content, not only whether the server returns success. A generic HTML fallback can return 200 at a missing Markdown URL. Also check whether a redirect has moved a link into a different version or language.
If your publishing system generates both HTML and Markdown views, compare one changed section in each. Confirm that warnings, prerequisites and code examples survive the conversion. Correcting a broken export is more useful than adding more links to that export.
Example: keep an API migration unambiguous
Suppose a fictional library moves from version two to version three. Its main documentation describes the new authentication parameter, but llms.txt still sends readers to the version-two quick start without labeling it as older material.
During the release review, update the primary quick-start link, label the retained version-two reference, and add a concise migration destination if one actually exists. Check the final documents to ensure their examples agree with their version labels. Do not rewrite the older reference to look current if users still need that historical version.
The success condition is straightforward: starting from the file, a reviewer can reach the correct current instructions and distinguish the supported legacy path. No ranking claim is needed to make that improvement useful.
Add the file to an existing release check
Use the documentation owner’s current checklist rather than creating an independent publishing process. Check changed destinations after moves, version releases and documentation retirements. Save the previous file alongside the release history so an incorrect link selection can be reverted.
The llms.txt Checker provides limited availability and title-heading observations. It does not validate every linked document or confirm that a provider used the file. Manually inspect representative linked content and use existing link checks where available.
If logs show requests to the file, classify them using the crawler verification guide. A fetch of llms.txt is not evidence that the linked pages were used in an answer.
Frequently asked questions
Should the file include private documentation URLs?
Do not publish sensitive paths or credentials. Keep private documentation behind its existing authorization boundary and maintain public references only for material intended to be public.
Does a missing file lower AI Crawl Score?
No. The optional llms.txt observation does not lower the score when the file is missing. Read the visibility explanation before treating it as a search requirement.