Most advice about AI crawlers comes down to two buttons: block everything, or allow everything. Both are wrong, and for the same reason. They treat every bot as identical when they are nothing alike. There is a better way, and it is not a bigger wall. It is knowing exactly who is at the door.
Put Disallow: / in your robots.txt and the well-behaved crawlers obey it, which means you have just removed yourself from the AI answers that fetch live pages and from the training data that teaches models you exist. Leave the door wide open instead, and you are feeding bulk scrapers and anything wearing a fake badge. The "training vs retrieval" split you usually hear is missing a category, and the missing one matters.
And cutting across all three: impostors, traffic claiming one of these names that the network evidence contradicts. An impostor is not an accusation of motive; it is a fact about identity. It is not the bot it says it is. So the real choice was never "block or allow". It is bait the search bots, keep the trainers, block the scrapers and impostors.
A single robots.txt directive cannot tell those apart, so it does collateral damage in both directions. Disallow: / turns away the Search bots that would have cited you and the Trainers that would have learned you. Allow-all lets the scrapers and impostors help themselves. Worse, your server logs cannot see the difference between a real PerplexityBot and a rented server wearing "PerplexityBot" in its user-agent. Acting on the name alone, you block your genuine referral sources while waving the fakes through. That is not control. That is firing blind.
The fix is to verify who each crawler actually is before you decide anything: is the IP inside the operator's published range, or does it pass a forward-confirmed reverse-DNS lookup? Either one verifies it; a contradiction marks it an impostor. This is where Unsourced fits, and the boundary is worth being precise about: Unsourced does not block anything for you. It labels every crawler (Verified, Impostor or Unverifiable, and Search, Trainer or Scraper) and hands you the exact, honest rule to act on in your own stack. The decision, and the switch, stay yours. You can check a single bot right now with the free AI crawler verifier.
Once you can see true identity, the three moves are simple:
This is not a config you set once and forget. It is a short loop: verify who is really crawling you, sort them by what they actually do, serve your best pages in a form the search bots can quote, and measure what changes. The goal is not to keep AI out. It is to be deliberately open with the crawlers that pay you back, and deliberately closed only to the ones that do not, or that were lying about who they are. When you want the full picture, every AI crawler labelled and sorted, that is what our verified crawler data is built on.
Put Disallow: / in robots.txt and the well-behaved crawlers obey it, which means you have removed yourself from the AI answers that fetch live pages and from the training data that teaches models you exist. Blocking the trainers feels productive, but being learned today is how a model comes to name you tomorrow.
AI Search bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) that fetch live pages to answer now and can cite and link you; AI Trainer bots (GPTBot, ClaudeBot, CCBot) that learn from you and name you later; and Scrapers (Bytespider, Diffbot) that do bulk extraction with no citation path back. Cutting across all three are impostors: traffic wearing a name the network evidence contradicts.
Bait your Search bots by giving them clean, structured, quotable data on the pages they fetch. Keep your Trainers; their value is delayed, not absent. Block scrapers and impostors, which have no citation upside or were never the real crawler. Because you verified identity first, you block the right targets rather than your own referral sources.
No. It is an observation layer. It labels every crawler that hits you (Verified, Impostor or Unverifiable, and Search, Trainer or Scraper) and hands you the exact rule to act on in your own stack. If you want enforcement, that is an optional Cloudflare Worker rule you turn on yourself, off by default, and it fails open so it can never wall off a real crawler.
How to verify identity before you decide anything.
Read it →Check a bot name or IP on the spot.
Read it →Verified data on who is really at your door.
Read it →Unsourced captures which AI assistants cite you, proves which crawlers really fetched your pages, and re-checks after you act — evidence, not a score.
© Unsourced — the evidence layer for AI search.