Verification starts with the real client IP
An AI crawler is verified only when the request’s trusted source address matches a current method published by the claimed operator. The User-Agent is useful for selecting candidates, but anyone can copy it. Keep three states in the data: claimed, verified, and unknown. “Unknown” means the available evidence could not establish identity; it does not mean malicious.
The first task is finding the address that actually connected to your edge. If the origin sits behind a CDN or reverse proxy, the socket peer will usually be the proxy. Accept a vendor-specific visitor-IP header only when the request came through a proxy you control and its configuration strips any client-supplied copy. Never trust the leftmost value of an arbitrary X-Forwarded-For chain.
Choose the operator’s published method
There are two common verification patterns, and providers may support one or both.
Published IP ranges: download the operator’s current JSON or documented list, parse each CIDR, and test whether the trusted client IP falls inside one of those networks. Preserve the source URL, retrieval time, and list version or hash. Perplexity publishes separate ranges for PerplexityBot and Perplexity-User in its crawler documentation, so membership in one list must not verify the other identity.
Reverse and forward DNS: Google’s request-verification procedure instructs operators to reverse-resolve the source IP, validate the returned hostname against an approved Google domain, then forward-resolve that hostname and require the original IP to appear. Check a real label boundary: crawl-1-2-3-4.googlebot.com can qualify, while googlebot.com.attacker.example and googlebot.example.com cannot. Reverse DNS alone is insufficient because an address owner can choose a misleading PTR record.
Google also publishes CIDR lists by crawler category. Use the list corresponding to common crawlers, special crawlers, or user-triggered fetchers rather than treating every Google network as Googlebot.
Build a reproducible verification record
For each sampled request, retain UTC timestamp, trusted client IP, full User-Agent, host, method, normalized path class, response status, edge action, verification method, evidence version, and result. Hash or redact sensitive query strings before storage. Cache successful network checks briefly to avoid a DNS lookup on every request, but expire them so provider changes are picked up.
Handle failures conservatively. A DNS timeout, stale range fetch, malformed address, missing proxy provenance, or forward mismatch produces unknown. Do not silently promote an old positive result after its evidence expires. The AI crawler log analysis guide shows how to aggregate these fields without converting claims into facts.
The AI Crawler Checker tests public policy and response behavior from this service. It cannot see the visitor IP in your private logs and therefore cannot verify historical traffic.
Explicitly hypothetical investigation
Assume a fictional publisher sees 9,200 requests labelled Googlebot during one day. Its origin log records only CDN addresses, but the edge export contains a signed client_ip field. The analyst rejects origin IPs and extracts 100 requests across hours and route classes from the trusted edge field.
For 73 requests, reverse DNS ends in googlebot.com, forward DNS returns the original IP, and the addresses also fall inside Google’s current common-crawler ranges. Seventeen have no matching forward result; ten cannot be checked because the signed field is missing. The report labels 73 verified and 27 unknown. It does not extrapolate that every one of the 9,200 requests was Googlebot. A WAF exception, if needed, uses the current range category plus route constraints rather than the User-Agent alone.
FAQ
Is a matching crawler User-Agent enough?
No. It identifies a claim. Verify the trusted source IP with the provider’s current published method.
Should failed DNS verification be labelled fake?
No. Record it as unknown unless other evidence establishes spoofing or abuse.