If you feel like the system is rigged against your bottom line, you are not imagining it, and for the first time the law is starting to agree. But a right you cannot enforce is a slogan, and every one of these fights runs on the same fuel: evidence of who took what, and when.
Publishers are not just complaining any more; they are filing, and regulators are acting. Penske Media, owner of Rolling Stone and Billboard, sued Google over AI Overviews, arguing the summaries cannibalise the clicks its journalism earns. In the UK, the Competition and Markets Authority has gone furthest of all: having designated Google with strategic market status, in mid-2026 it secured a world-first requirement that publishers can refuse to have their work fed into Google's AI answers without being quietly demoted in ordinary search (the CMA's own summary is here).
Read that twice. The right to say "use my work in your AI and I will walk" just became real. But a right you cannot enforce is a slogan, and every one of these fights runs on the same fuel: evidence of who took what, and when. That is exactly where the bots have stopped playing fair.
When a crawler hits your server it introduces itself with a User-Agent string, a single line of text it writes about itself. A genuine OpenAI crawler says GPTBot. Google's says Googlebot. Your firewall and robots.txt make their decisions off that string.
Here is the hole. Nobody dares block Googlebot, so almost every server on the web waves it straight through. Scrapers and data brokers know this, so they do not break your security. They pin on a name tag that says Googlebot and walk in through the front door you are holding open. Your content leaves to train a commercial model that will never send you a visitor, and because the legacy analytics stack only fires when a human browser loads a page, nobody in the building ever sees it happen.
Trusting a User-Agent is trusting a man because his t-shirt says "Security". It is self-reported. It proves nothing. You cannot ask a bot who it is. You verify it.
Identity at the network level is far harder to fake than a line of text. Two checks do the heavy lifting, and neither is magic.
Forward-confirmed reverse DNS (FCrDNS). Take the IP the crawler connected from, look up the name that IP points back to, then resolve that name forward again and confirm it lands on the same IP. An attacker can rename their user-agent for free, but cannot forge the reverse-DNS record for an IP they do not own, and cannot hold a real connection open from a source IP they cannot route. That mismatch is what catches them. It is a consistency check, not cryptography; anyone selling it as an un-spoofable cryptographic handshake is overselling, and it does not need the hype to work.
Published IP-range verification. Several major operators, OpenAI and Google among them, publish the exact IP ranges their crawlers run from. A bot announcing itself as GPTBot from a discount cloud block in the wrong region is not GPTBot. It is the same approach Google itself documents for verifying its crawler.
Run both and your logs stop being a guess. "Googlebot hit us 4,000 times this week" resolves into "1,200 verified Google, 2,800 impostors wearing the name". That second number is the one almost nobody can show you, because most tools never get past reading robots.txt. You can check a single bot yourself with our free AI crawler verifier.
Whether you plan to block the scrapers, exercise an opt-out the moment it lands, or join a publisher action, you will be asked the same question: prove it was taken from you. A robots.txt directive proves what you asked. It says nothing about who ignored you. The proof lives in verified server logs, forensic telemetry that ties a specific extraction to a specific, identity-checked actor on a specific date. That is the gap between "we think AI bots are scraping us" and an evidence chain you could take to a solicitor. The day you decide to act, you want the receipts already in hand, and the sane way to route each crawler is to bait, keep or block by verified identity rather than blanket-block.
The web stopped being a polite distribution network and became a contested data supply chain. You do not get a vote on the macro shift. You do get to decide who extracts from your servers, and, when it counts, whether you can prove it. Stop trusting the t-shirt. Demand proof of identity.
Because the bot name is a User-Agent string the visitor writes about itself. Nobody dares block Googlebot, so almost every server waves it through. Scrapers know this, so they pin on a Googlebot name tag and walk in through the door you are holding open. Your firewall and robots.txt make their decisions off a line of text that proves nothing.
At the network level, which is far harder to fake than a string. Two checks do the work: forward-confirmed reverse DNS (resolve the IP to a hostname, then resolve that hostname back and confirm it lands on the same IP and belongs to the operator) and published IP-range verification. An attacker can rename their user-agent for free, but cannot forge the reverse-DNS of an IP they do not own.
No, and anyone selling it as an "un-spoofable cryptographic handshake" is overselling. It is a consistency check that rests on something a spoofer cannot control: the reverse DNS of an IP they do not route. It does not need the hype to work.
Whether you plan to block scrapers, use a new opt-out, or join a publisher action, you will be asked to prove content was taken from you. A robots.txt directive proves what you asked, not who ignored it. The proof lives in verified server logs that tie a specific extraction to a specific, identity-checked actor on a specific date.
Paste a bot name or IP and check it against published ranges and FCrDNS.
Read it →The three honest verdicts every AI visitor falls into.
Read it →We verified 9,543 AI-crawler visits. 5.4% were fakes.
Read it →Unsourced captures which AI assistants cite you, proves which crawlers really fetched your pages, and re-checks after you act — evidence, not a score.
© Unsourced — the evidence layer for AI search.