A path check catches crawlers asking for pages that do not exist. An identity check catches crawlers that are not who they say. On our own log, each caught something the other missed.
There are two ways to tell a real AI crawler from a fake one in your server logs. Ask who sent the request: is the IP inside the operator’s published ranges, and does its reverse DNS confirm it? Or ask what was requested: did the page ever exist? VantaWeb published the second method and invited a comparison. So we ran both tests on the same four weeks of our own log. They agree most of the time. Where they disagree, each one catches something the other misses.
In July, VantaWeb cleaned an eight-day log from its own site with a path test. Of 3,727 requests carrying an AI-crawler or search-engine user agent, 3,109 (83.4%) asked for paths the site had never published. VantaWeb published the method and the aggregates, and invited a comparison. Our own reports use a different test: identity. We check each visit’s IP against the operator’s published ranges and forward-confirmed reverse DNS. So we ran both on the same log: every visit to unsourced.app that claimed an AI-crawler name, 1 to 28 October 2026. That is 6,805 visits from 739 IP addresses, to 1,012 different paths.
For the path test, a path counted as published if it appears in any version of our sitemaps since launch, is a page the site serves (login and demo pages are linked but not in the sitemap), is a standard file such as robots.txt or llms.txt, or sits in a folder our pages load, such as built assets. Everything else counted as never published.
Genuine crawlers almost never ask for pages that do not exist. Of 2,502 visits the identity check proved, 2,480 (99%) asked for a real page. And visits that could not back up their crawler name mostly went fishing: 1,705 of 2,047 (83%) asked for paths we never published. On those 1,705 visits, both tests reach the same verdict independently.
Fake crawlers reading real pages. 342 visits used a crawler name their IP could not back up, yet asked only for real pages: the homepage, robots.txt, login and register pages, and our guides to individual AI crawlers. A path test would pass every one. These are the visits that matter most to a publisher, because a fake crawler reading your real content is a scraper borrowing a trusted name.
Real crawlers asking for odd paths. 22 proven visits asked for a path that was never published. 17 of them were requests for /research/robots.txt, robots.txt looked up inside a subfolder. A path test would flag these as junk. They are genuine crawlers making a harmless mistake.
Most of the unverified traffic came from one pattern. 1,578 visits came from nine Google Cloud customer servers. Each server rotated through nine or ten AI-crawler names, from GPTBot and ClaudeBot to PerplexityBot and CCBot, and asked for files no website should serve: /.env, /.env.production, and cloud credential paths. It is a credential scanner dressed as AI crawlers. Our site returned nothing sensitive to any of it. Leave that campaign out and 469 of the remaining 5,227 visits were unverified, about 9%.
Use both tests. A path test is cheap and needs nothing but your sitemap, and it clears out scanners fast. But it cannot see a fake crawler that reads real pages, and that is the case worth knowing about. An identity test catches those, and it is the only way to say a visit really came from OpenAI, Anthropic or Perplexity. Run identity checks against current IP lists: operators change their ranges, and old visits checked against today’s lists look fake when they were not.
This is one site, four weeks, and a site about AI crawlers, which likely draws more spoofing than most. 2,256 visits (33%) used crawler names with no published IP ranges, so identity could not rule either way. We make no claim about who sent any request. The aggregates are free to reuse with credit: CSV and JSON. VantaWeb’s data is at vantaweb.io/data.
An identity check asks who sent the request. It compares the IP with the ranges the operator publishes for its crawler, and with forward-confirmed reverse DNS. A path check asks what was requested: was that page ever published on the site? The first spots a crawler that is not who it claims to be. The second spots requests for pages that do not exist, which is what scanners do.
Rarely. Of 2,502 visits where the identity check proved the crawler, 22 asked for a path the site never published, under 1%. Most of those were requests for robots.txt inside a subfolder (/research/robots.txt), a known quirk of some crawlers resolving the file relative to a page.
Yes. Of 2,047 visits whose IP could not back up the crawler name they used, 342 asked only for real pages: the homepage, robots.txt, login and our crawler guides. A path check would count those as clean. Only the identity check caught them.
We make no claim about who sent any request. What we can show: 1,578 of the unverified visits came from nine Google Cloud customer servers. Each one rotated through nine or ten AI-crawler names and asked for secret files, such as /.env and cloud credential paths. That is a scanner wearing crawler names, not the AI companies it named.
14,227 visits across the sites we monitor, checked by identity.
Read it →How fake AI crawlers work, and how to spot one.
Read it →Check any IP against an AI operator’s published ranges.
Read it →Unsourced captures which AI assistants cite you, proves which crawlers really fetched your pages, and re-checks after you act — evidence, not a score.
© Unsourced, the evidence layer for AI search. Our data and findings are free to quote and cite — please attribute to Unsourced and link to unsourced.app.