Unsourced
Field data

AI crawlers exposed: what they really do on your site.

The bots that read your new pages first are feeding future models, not today’s answers. The bots behind citations read different pages. And in our logs, half the “AI bots” were not who they claimed.

Open your server logs and you will find AI bots every day. Ask ChatGPT or Perplexity about your field and you will rarely find your site cited. That is not a glitch. In our logs, half of those bots were fakes. The real ones that read your new pages first are feeding future models. The bots behind today’s citations are different bots, reading different pages. We traced it bot by bot through five months of logs, keeping only visits we could prove came from the real operator.

Published 10 November 2026 · Unsourced Research · All research

How we checked

We took every visit to unsourced.app that used an AI-bot name between 10 June and 10 November 2026. That came to 34,577 visits under 28 bot names, from 3,298 IP addresses. We checked each address against the IP list the operator publishes, and against forward-confirmed reverse DNS. Everything below uses only the visits that passed.

1. Half the “AI bots” are fakes

Only 17,575 of the 34,577 visits (51%) came from the operator they named. The rest failed the check, or used a name that cannot be checked. The difference shows in what they asked for. Genuine bots asked for secret files such as /.env 0 times. The others asked 5,105 times. Many ran on rented cloud servers. One fleet rotated through ten AI names on the same machines. If you judge AI bots by the name in the log, half of what you see is noise. Check any address with Verify a bot.

2. The bot that reads your content first is not the bot that cites you

We published 16 research posts from August onwards. GPTBot, OpenAI’s training crawler, fetched 15 of them within a day of the publish date, and the last within four days. ClaudeBot, Anthropic’s training crawler, fetched all 16, half of them within about 12 hours. OAI-SearchBot fetched 2 of the 16. That is the crawler OpenAI says surfaces sites in ChatGPT search. PerplexityBot found all 16, but took a median of 18 days.

unsourced.app Who read 16 new posts New posts each genuine bot fetched at least once, Aug to Nov 2026 GPTBot16 of 16 ClaudeBot16 of 16 PerplexityBot16 of 16 Amazonbot16 of 16 CCBot10 of 16 OAI-SearchBot2 of 16 Genuine visits only. GPTBot median 19.5 hours from midnight UTC on the publish date. PerplexityBot median 18 days. unsourced.app.

Training crawlers feed future models. A model only knows what was in its training data, so a fetch today may shape an answer months from now, or never. Citations today come from engines that search the live web. In our own citation tracking, every citation came from one of those: Perplexity, ChatGPT search or Bing. None came from a model answering from memory. And they cited reference pages that answer one question, such as our guide to GPTBot, not the new posts the training crawlers rushed to read. We made the wider case in a crawl is not a citation. These logs show it bot by bot.

3. robots.txt is read in ways the docs barely mention

unsourced.app Who reads robots.txt robots.txt requests by genuine bots, 10 Jun to 10 Nov 2026 ClaudeBot1,219 OAI-SearchBot782 PerplexityBot186 GPTBot0 Amazonbot0 Genuine visits only: GPTBot 1,577 visits, Amazonbot 1,537 visits. unsourced.app.

GPTBot never asked for the robots.txt file. In 1,577 genuine visits, GPTBot requested robots.txt 0 times. That looks like a crawler ignoring the rules, but it is probably not. OpenAI’s crawler docs say that if a site allows both of its bots, “we may use the results from just one crawl for both”. OAI-SearchBot read our robots.txt 782 times, and 415 of those reads came from addresses on both bots’ IP lists. We cannot see inside OpenAI, but the pattern fits one shared read.

Amazonbot never asked either. It made 1,537 genuine visits and 0 robots.txt requests. Amazon’s Amazonbot page, updated 26 October 2026, says it fetches robots.txt or uses a copy cached within the last 30 days. Over five months, that implies at least five fetches. We logged none under the Amazonbot name. We cannot explain the gap.

ClaudeBot asks constantly. It read robots.txt 1,219 times, about 10 times on each day it visited. 80% of its visits were robots.txt or a sitemap, and only 9% were pages. Anthropic says ClaudeBot honours robots.txt and supports Crawl-delay. In practice, it checks the rules far more often than it reads the content.

Two famous names are not crawlers at all. Google’s crawler list says Google-Extended has no separate user agent string. Apple’s Applebot page says Applebot-Extended does not crawl webpages. Both are robots.txt tokens. Yet we logged 474 visits named Google-Extended and 9 named Applebot-Extended. None came from Google or Apple. If you see either name in a log, the visit is not what it claims.

4. Some AI crawlers ran our JavaScript

A common belief is that AI crawlers never run JavaScript. Our pages load a small script that, when it runs, pings one URL. Nothing else asks for that URL. Genuine OAI-SearchBot sent the ping 199 times, in every month from June to October. GPTBot sent it 18 times and ClaudeBot 7. So these crawlers ran our script at least some of the time. There is a twist. The ping URL sits under /api/, which our robots.txt told OAI-SearchBot to skip until 22 October. Running the page’s script fetched a URL the rules disallowed.

Keep it in proportion: 199 pings is small next to OAI-SearchBot’s 2,165 genuine visits. None of the AI operators we checked documents JavaScript rendering for its AI crawlers. Only Apple says its search crawler, Applebot, may render pages. Put your key content in the HTML.

Documented, observed, unknown

Documented by the operator (checked 10 November 2026). OpenAI may use one crawl for both GPTBot and OAI-SearchBot, and robots.txt changes take about 24 hours. Perplexity-User generally ignores robots.txt. Meta-ExternalFetcher may bypass it. Amazonbot may use a robots.txt copy up to 30 days old. DuckAssistBot takes 72 hours to act on a Disallow.

Seen in our logs, not in any doc. GPTBot’s next-day rhythm on new posts. ClaudeBot’s constant robots.txt reads. OAI-SearchBot rarely fetching new posts. Script pings from three AI crawlers. No robots.txt reads at all from GPTBot or Amazonbot.

Unknown or undisclosed. How any engine picks which fetched page to cite. Bytespider: we found no official ByteDance page, and its reverse DNS did not confirm, so its visits stay unverified. Meta publishes no IP list or reverse DNS method, so no Meta visit can be verified. Cohere, You.com and Huawei’s PetalBot had no official crawler page we could load on 10 November.

What to do about it

Verify before you trust your logs. Half the AI-bot visits here were not from the operator. Our guide to verifying before you block covers the method.

Know which bot you are blocking. Training crawlers and search crawlers do different jobs. Blocking one does not block the other. More in blocking AI crawlers is not one switch.

Write pages that answer one question. The pages that earned citations were reference pages, each about one bot or one engine. Opinion posts did not.

Don’t judge AI visibility by a prompt score alone. This is our opinion. A prompt score asks an engine a set of questions and counts the mentions. It is a snapshot, not a cause. Engines change about half their sources between runs of the same question, so one score can swing for no reason. Logs show which bots read which pages. Verification shows which of those bots are real. A score without that context cannot tell you what to fix. More in why an AI visibility score is a guess and AI cites different sources each time.

Further reading

This post ties together our earlier crawler research: how AI crawler spoofing works, one in five AI crawlers was an impostor, why robots.txt cannot stop AI scrapers, AI is being shown a different website and identity or path.

Limits

This is one site over five months, and a site about AI crawlers draws more fakes than most. We checked identity against today’s IP lists and live DNS. Operators change their ranges, so a few older visits may be judged wrongly. Bot names with no way to verify them are left out. Googlebot and Applebot are not on our tracking list, so Google and Apple search crawling is out of scope. Time to first fetch is counted from midnight UTC on the publish date.

Common questions

Why do AI bots crawl my site but AI answers never cite it?

Because the busiest AI bots are training crawlers. They feed future models, not today’s answers. In our tracking, every citation came from an engine that searches the live web, such as Perplexity, ChatGPT search or Bing. Those engines rely on search crawlers. On our site, OpenAI’s search crawler fetched only 2 of 16 new posts, while its training crawler fetched all 16.

How many AI bots in my logs are fake?

On our site, about half. Of 34,577 visits that used an AI-bot name over five months, 17,575 (51%) came from the operator they named. The genuine bots never asked for secret files such as /.env. The others did 5,105 times. Check any IP against the operator’s published list before you trust the name.

Does GPTBot read robots.txt?

Not under its own name, on our site. Genuine GPTBot made 1,577 visits in five months and asked for robots.txt 0 times. OpenAI says that when a site allows both GPTBot and OAI-SearchBot, it may use one crawl for both. OAI-SearchBot read our robots.txt 782 times, 415 of them from addresses on both bots’ published IP lists.

Do AI crawlers run JavaScript?

Some did, sometimes. Our pages load a small script that pings one URL when it runs. Genuine OAI-SearchBot sent that ping 199 times between June and October. GPTBot sent it 18 times and ClaudeBot 7. None of the AI operators we checked documents this, so keep your key content in the HTML.

Is Google-Extended a crawler?

No. Google says Google-Extended has no separate user agent string. It is a robots.txt token that controls how Google uses crawled content for Gemini. Apple says Applebot-Extended does not crawl webpages either. We logged 474 visits named Google-Extended and 9 named Applebot-Extended. None came from Google or Apple.

Related

A crawl is not a citation

Why a bot fetch and an AI citation are different things.

Read it →

Identity or path?

Two ways to clean an AI-crawler log, run on one log.

Read it →

Verify a bot

Check any IP against an AI operator’s published ranges.

Read it →

See the evidence for your own site

Unsourced captures which AI assistants cite you, proves which crawlers really fetched your pages, and re-checks after you act: evidence, not a score.

© Unsourced, the evidence layer for AI search. Our data and findings are free to quote and cite. Please attribute to Unsourced and link to unsourced.app.

Crawler test: this link is blocked in our robots.txt.