ScraperDiffbot
Last verified: 2026-07-16 · maintained by Unsourced
Diffbot extracts structured data from pages for its knowledge graph, sold on to third parties. It offers no direct citation or referral benefit to the sites it crawls.
Diffbot reads pages to extract structured facts into a commercial knowledge graph it licenses to other companies. Your content becomes a data point in a product sold onward, with no link, mention or traffic flowing back to you. Sites that see no value in feeding a resold dataset commonly block it; those who happen to license Diffbot themselves may weigh it differently.
Diffbot publishes neither an IP-range feed nor a reverse-DNS footprint for Diffbot, so there is no way to confirm it by network identity. Treat every request carrying the Diffbot user-agent as an unverified claim, and judge it on what it does rather than the name it gives.
Recommended: block. Diffbot extracts your content into a commercial knowledge graph it licenses to others, with no link, mention or traffic flowing back — so unless you license Diffbot yourself, there is little reason to feed it.
If you do choose to act in robots.txt (which crawlers honour but don't enforce):
User-agent: Diffbot Disallow: /
What does Diffbot do with my content?
It extracts structured facts into a commercial knowledge graph that Diffbot licenses to other companies — your page becomes a data point in a product sold onward, with no link or traffic flowing back.
Is there any reason to allow Diffbot?
Mainly if you license Diffbot's knowledge graph yourself and want your own content represented in it. Otherwise there's little upside, which is why sites that don't use it commonly block it.
Can I verify Diffbot by IP?
No — Diffbot publishes neither an IP-range feed nor a reverse-DNS host, so a request carrying its name can't be confirmed at the network level.
Unsourced tells you which requests claiming to be Diffbot are verified, which are impostors, and which AI assistants actually link to your pages.
Check your site free →10-day free trial · no card required · cancel anytime