On September 15, Cloudflare begins blocking AI crawlers by default on pages that carry ads, and pushing AI companies to separate search crawlers from training and agent crawlers or be blocked. Overnight, block, charge, or allow becomes a decision every site owner has to make. The problem sits underneath all three options: the user-agent you would base the decision on is a claim, and a large share of the claims are false.
Cloudflare is turning on default blocking for AI crawlers on pages that carry ads, and telling AI companies to separate the crawlers they use for search from the ones they use for training and agents (Cloudflare's own announcement; TechCrunch coverage). A crawler that blends the two risks being blocked by default across a large slice of the web. Alongside it, Pay-Per-Crawl lets a publisher answer a crawler with an HTTP 402 and a price, so access becomes something a crawler pays for rather than takes. The three choices — block it, charge it, or allow it — now land on every site owner at once.
Block, charge, allow: all three are keyed on identity. And identity, as most rules define it, is the user-agent string — the one piece of the request the sender writes themselves. Over 30 days of our own traffic, more than one in five requests wearing an AI crawler's name failed verification against the operator's published IP ranges and reverse DNS. The same addresses turned up claiming to be five different AI companies at once. The full data is in our impostor study. Decide by the header alone and you get the worst of both directions: you block the real crawler that could have cited you, and you wave through the impostor wearing its name from someone else's server.
The instinct to block assumes that no crawl means no mention. It does not. In a 2026 analysis of Google AI Mode citations, 27 of the 52 cited domains — over half — were sites that block at least one AI crawler in their robots.txt, roughly three and a half times the rate across the sample as a whole. A citation can be drawn from training data, the open index, or a third-party page that quotes you, none of which depend on today's fetch. Blocking controls what gets trained on and fetched. It does not control whether an assistant names you.
It is also worth being honest about what a block even does. robots.txt is honoured, not enforced. Well-behaved crawlers obey it; the rest read straight past it. In one 2026 sample about 40% of GPTBot bans were simply ignored by the crawler they were meant to stop. A rule that only works on the bots that agree to follow it is a preference, not a control.
So the answer to Cloudflare's block-charge-allow question is not a blanket setting. It is two steps, in order. First, verify identity — match the source IP to the operator's published range and forward-confirm it by reverse DNS, so you are acting on the crawler that is really there, not the name it typed. Then decide by what that verified crawler does with your content: court the ones that cite and link, keep the trainers if you want to be in the next model, and block the extractors and the proven impostors. Our crawler directory sets out where each major AI crawler lands and how to confirm it.
If you do enforce, enforce at the edge, where a request can actually be refused, and only refuse identities you can disprove — an AI-crawler name arriving from outside that operator's published range. That blocks the impostors without ever risking the real crawler, which is the only version of blocking that does not quietly cost you citations.
This is the part Unsourced does for you. An optional edge worker sits in front of your own domain and turns away the spoofed crawlers — an AI-crawler name from outside that operator's published range — while letting the verified official ones straight through. It stays off until you switch it on, only ever refuses a request whose claimed identity it can disprove, and fails open, so a genuine crawler is never at risk. And whether or not you block, it records which verified crawler fetched which of your pages, so you keep a running log of what AI has actually ingested from your site rather than a user-agent you chose to trust. You can also check any single request against published ranges and forward-confirmed reverse DNS with our free crawler verifier.
September 15 forces a decision, but the decision is only as good as the identity behind it. Block, charge, and allow all fall apart the moment the name in the header is a lie — and in our data, more than one in five of them were. Prove the crawler is real first. Everything else is a rule written on trust you cannot verify.
Cloudflare starts default-blocking mixed-use AI crawlers on ad-bearing pages, and gives AI companies a deadline to split search crawlers from training and agent crawlers or risk being blocked across many publishers. Publishers can also charge crawlers through Pay-Per-Crawl, which answers a request with an HTTP 402 until the crawler agrees to pay.
Not necessarily. In a 2026 analysis of Google AI Mode citations, 27 of 52 cited domains — over half — were sites that block at least one AI crawler in robots.txt, about three and a half times the baseline rate. A citation can come from training data, the open index, or third-party pages, not only a live crawl. Blocking controls fetching, not whether you are named.
Check the source IP against the operator's published ranges and confirm it with forward-confirmed reverse DNS. In our own 30 days of traffic, more than one in five requests wearing a major AI crawler's name came from outside that operator's ranges. The user-agent alone proves nothing.
Only the ones that choose to obey it. robots.txt is honoured, not enforced. In one 2026 sample about 40% of GPTBot bans were ignored by the crawler they targeted. Enforcement that actually holds happens at the network edge, and only against identities you can disprove.
Every major AI crawler: who it is, how to verify it, and whether to allow or block it.
Read it →Check whether a bot name or IP is really who it claims.
Read it →Our 30-day data: 21.8% of AI-crawler hits failed verification.
Read it →Unsourced captures which AI assistants cite you, proves which crawlers really fetched your pages, and re-checks after you act — evidence, not a score.
© Unsourced, the evidence layer for AI search. Our data and findings are free to quote and cite — please attribute to Unsourced and link to unsourced.app.