ScraperByteDance
Last verified: 2026-07-16 · maintained by Unsourced
Bytespider is ByteDance's crawler, widely observed fetching aggressively and inconsistently honouring robots.txt. With no citation or referral benefit, many sites choose to block it.
Bytespider is ByteDance's crawler, widely reported fetching at high volume and honouring robots.txt only inconsistently. Operators tend to notice it through the load it adds rather than any traffic it returns, since there's no citation or referral path back to crawled sites. That cost-without-benefit profile — plus the patchy robots compliance, which means you should verify any block actually holds — is why blocking it is a common call.
ByteDance publishes no machine-readable IP-range feed for Bytespider, so the dependable check is reverse DNS. A genuine request's IP reverse-resolves to bytedance.com, and that hostname forward-resolves back to the same IP — the round trip is what confirms it. A Bytespider user-agent from an address that won't forward-confirm is an impostor.
Recommended: block. Bytespider fetches at high volume for ByteDance and honours robots.txt only inconsistently, returning no citation or referral — so blocking is the common call, but confirm the block actually holds (its requests reverse-resolve to bytedance.com), since the worst offenders ignore the rule.
If you do choose to act in robots.txt (which crawlers honour but don't enforce):
User-agent: Bytespider Disallow: /
Does Bytespider actually obey robots.txt?
Only inconsistently, by widespread reports — so if you choose to block it, confirm the block is actually holding rather than assume the directive is respected.
Why do so many sites block Bytespider?
Because it's been observed fetching at high volume for ByteDance with no citation or referral in return — operators tend to notice it through the load it adds, not the traffic it sends back.
How do I tell a real Bytespider from a spoof?
By reverse DNS — a genuine request resolves to bytedance.com. ByteDance publishes no IP-range feed, so the reverse-DNS round trip is the check that catches an impostor.
Unsourced checks each crawler against published ranges andreverse DNS, and shows where AI search cites you instead of a competitor.
Check your site free →10-day free trial · no card required · cancel anytime