A bot fetching your page and an AI answer citing your page are two different events. For two years the industry has talked about them as if they were one. Cloudflare just built a payment model on the difference, and it changes how you should read every AI crawler that touches your site.
Being "visible in AI search" is really three separate things. They happen at different times, for different reasons:
Most "AI visibility" advice collapses all three into one number. That is where the thinking goes wrong, because a crawl does not guarantee a citation, and a citation does not guarantee a referral. Each one has to be earned, and measured, separately.
Cloudflare is evolving its Pay Per Crawl programme into Pay Per Use, which lets publishers charge AI companies when their content creates value, for example when it appears in an AI-generated answer, not simply when a bot fetches the page. They moved the meter from the crawl to the use, because the crawl was never where the value was.
Their own data shows how far the two have drifted apart. In Cloudflare's analysis, as of June 2025 Google crawled about 14 pages for every referral it sent back. For AI companies the gap was orders of magnitude wider: OpenAI's crawl-to-referral ratio was 1,700 to 1, and Anthropic's was 73,000 to 1. A crawler can take thousands of pages and send almost nobody back. Paying on the crawl would reward exactly the wrong behaviour. Paying on the use lines the payment up with the value.
The moment payment depends on use rather than fetch, "did an AI use my content" stops being a curiosity and becomes an accounting question. And you cannot answer an accounting question with a ranking or a robots.txt file.
The first-party consoles help, but only partly. Google's Search Console now has an AI-search performance report, and Bing Webmaster has an AI Performance view. Both are worth having. Neither shows you the crawl fetch itself, neither reaches across to the other engines or to ChatGPT, Perplexity and Claude, and neither ties a citation back to the specific bot visit that produced it. To settle who used what, and when, you need the crawl and the citation recorded together, on your side.
The chain is short, and it is checkable:
This is the whole reason Unsourced exists. We are the evidence layer for AI search: we prove which crawlers really reached you and which AI assistants cite you, and we keep the two linked. For the current picture of who is really crawling real sites, verified against published ranges and reverse DNS, see our AI Crawler Report.
From 15 September, Cloudflare blocks mixed-use AI crawlers on ad-supported pages by default. Blocking sounds safe, but it is only safe if you know which crawlers earn you citations and which just take. Block a search bot that cites and links its sources and you have handed away a citation to keep out a scraper. Verify identity first, then decide. We wrote the full playbook on that in bait, keep, or block.
The short version: a crawl is not a citation, a citation is not a referral, and the gap between them is now where the money is. If you cannot tell the three apart in your own data, you cannot prove what you are owed.
A crawl is an AI bot fetching your page. A citation is an AI answer using or naming your content when someone asks a question. A crawl can happen thousands of times without ever producing a citation, and a citation can be built from a page crawled weeks earlier. They are separate events, worth measuring separately.
It means Cloudflare's model now pays on use rather than on crawl, which is the right place to put the meter. Whether any given AI company pays, and how much, is still being worked out. What you can do today is record which AI actually used your content, so that whenever the payment question is settled you already hold the evidence.
Not necessarily. Being crawled is necessary but not sufficient. AI answers lean toward sources they can corroborate and that carry authority for the specific question, so plenty of crawled pages are never cited. More crawl access is a leading signal; citations are the lagging outcome.
Verify each crawler against published IP ranges and reverse DNS, log which verified bot fetched which page, then match those fetches to the AI answers that later used your wording. That pairing is what turns "we think AI reads us" into evidence. It is what our <a href="/methodology">methodology</a> is built to produce.
Verified data on who is really at your door.
Read it →Route AI crawlers by verified identity, not by the header.
Read it →Check a bot name or IP on the spot.
Read it →Unsourced captures which AI assistants cite you, proves which crawlers really fetched your pages, and re-checks after you act — evidence, not a score.
© Unsourced, the evidence layer for AI search. Our data and findings are free to quote and cite — please attribute to Unsourced and link to unsourced.app.