Control discovery and user fetches separately
For Perplexity, evaluate PerplexityBot and Perplexity-User as different access paths. Perplexity’s official crawler documentation says PerplexityBot is used to surface and link sites in search results and is not used to crawl content for foundation-model training. Perplexity-User supports requests initiated by users, is not a general crawler, and generally ignores robots.txt because the user requested the fetch.
That last distinction changes the security review: robots policy can express a preference for PerplexityBot, but sensitive URLs must reject unauthorized requests regardless of User-Agent.
Review policy, network identity, and page response
Choose one representative URL per route class: article, product, faceted listing, cart, account, preview, and API. Write the desired outcome for both identities before changing controls.
For PerplexityBot, evaluate the actual robots group and path. For Perplexity-User, focus on whether the public/private boundary is correct. Then:
- Fetch the final public URL through the normal CDN path. Record redirects, response status, canonical, robots meta, and the visible answer in raw HTML.
- Confirm carts, accounts, drafts, signed downloads, and administrative routes require real authorization. A
Disallowline does not secure them. - Inspect WAF decisions separately. Perplexity publishes different JSON IP lists for PerplexityBot and Perplexity-User and recommends combining the current IP ranges with the corresponding User-Agent in WAF rules. Retrieve those ranges from the official endpoints referenced in its crawler documentation rather than copying a permanent list into a ticket.
- Scope any allow rule to the correct identity, current ranges, public host, and required routes. Do not exempt all requests containing the word “Perplexity.”
- Retest an allowed public page and a denied private page, then save the exact configuration and timestamp.
The AI Crawler Checker helps inspect a public response. The robots.txt AI Checker helps explain which rule matches PerplexityBot. These checks do not create traffic history or confirm a real Perplexity request.
Interpret the result carefully
A successful test means the tested request reached the tested content at that time. It does not establish that Perplexity indexed the URL, selected it as a source, or will cite it. Conversely, a lack of observed referral sessions does not prove that the crawler never visited; crawler logs and visitor analytics measure different events.
Keep provider identity fields specific in logs: claimed token, trusted client IP, range-list version, verification result, route, and edge action. If the CDN masks the visitor address and you cannot recover it from a trusted field, classify the request as unknown rather than verified.
Hypothetical commerce test
Consider a fictional retailer, Lantern Supply. It wants product detail pages discoverable, but /search is expensive and /cart is private. The team allows PerplexityBot on /products/, disallows /search, and retains authentication plus session checks on cart and account routes.
An edge log shows a PerplexityBot User-Agent receiving 403 on a product. The request’s trusted visitor IP belongs to the current official PerplexityBot range, and the WAF matched an old “unknown automation” rule. The team adds a narrowly scoped exception for verified PerplexityBot traffic to cached product pages. It does not exempt Perplexity-User, checkout, or search. Retesting returns 200 for the product and 401 for an unauthenticated account request.
This hypothetical outcome demonstrates corrected access on two routes. It does not show that the product appeared in an answer or generated a sale.
FAQ
Is PerplexityBot documented as a training crawler?
No. Perplexity describes it as a search crawler and explicitly separates it from foundation-model training collection.
Can I rely on robots.txt to block Perplexity-User from private pages?
No. Protect private pages with authentication and authorization for every client.