Applebot-Extended is a use control
Applebot-Extended does not crawl webpages. Apple says it is a secondary robots token used to decide how content already crawled by Applebot may be used for training Apple’s generative foundation models. A page disallowed for Applebot-Extended can still be crawled by Applebot and appear in Apple search experiences. These distinctions are documented in Apple’s Applebot support page.
Treat Applebot as the discovery crawler and Applebot-Extended as a data-use preference. Review both, because changing one does not imply the desired outcome for the other.
Account for Apple’s fallback rule
Apple documents an unusual compatibility behavior: if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot directives. This can make a Google Search policy an accidental Apple discovery policy.
Use a path matrix to make the result explicit:
- Fetch the live robots file on each hostname. Record groups for
Applebot,Applebot-Extended,Googlebot, and*. - Evaluate representative public and restricted paths for Applebot. If there is no Applebot group, apply Apple’s documented Googlebot fallback during the review.
- Evaluate the same paths for Applebot-Extended as a separate preference. Do not look for Applebot-Extended in access logs because Apple states that it does not crawl pages.
- Request an Applebot-allowed public page through the production edge. Check redirects, final status, canonical, robots meta, and whether the main answer is usable without blocked resources. Apple notes that Applebot may render pages and that blocking required CSS, JavaScript, XHR, or images can impair rendering.
- Protect account, preview, and paid-content routes with authentication. Robots rules publish preferences and paths; they do not enforce confidentiality.
- Save the tested file and time, deploy the smallest change, and fetch the live file again after caches have updated.
The robots.txt AI Checker is useful for the explicit product-token rules. Document the Googlebot fallback in the review notes because a generic parser may not model provider-specific behavior. The AI Crawler Checker can inspect a public response but cannot prove an Applebot visit.
Separate indexing controls from Extended
Applebot also supports page-level indexing directives such as noindex, nosnippet, and nofollow, according to the same official page. Those directives affect search behavior and presentation; Applebot-Extended is about the described foundation-model use. A team that wants Apple discovery but no Extended use should keep the page crawlable and indexable for Applebot while expressing the separate Extended restriction.
Do not infer results from silence in analytics. Applebot crawling, Apple search appearance, generative-model use, and human referral sessions are different events with different evidence.
Hypothetical travel-site review
Imagine a fictional travel publisher, Coastline Atlas. Its robots file has a Googlebot group that disallows /maps/private/, but no Applebot group. The editorial team assumes Applebot can crawl every destination page.
During a release review, the team applies Apple’s fallback and discovers that Applebot inherits the Googlebot restriction. That is correct for private draft maps, but a public guide at /maps/private-islands is unintentionally caught by the path prefix. The team adds an explicit Applebot group that allows public destination guides and disallows the true draft directory. It also disallows Applebot-Extended across the site to express its training preference.
After deployment, the public guide returns 200 with a self-canonical and accessible raw text; the draft endpoint still requires login. The recorded result is a corrected Applebot policy and a separate Extended choice. It is not evidence that Apple indexed the guide or changed its placement.
FAQ
Does Applebot-Extended block Applebot crawling?
No. Applebot-Extended does not crawl pages; Applebot can remain allowed for discovery.
What happens when robots.txt names Googlebot but not Applebot?
Apple says Applebot follows the Googlebot directives, so include that fallback in the audit.