What CCBot does
CCBot builds Common Crawl's public web archive — the dataset many AI labs (and researchers) train on. Blocking it removes you from that public corpus, which indirectly affects many models at once rather than one vendor's.
Fact sheet
- Operator
- Common Crawl
- robots.txt token
- CCBot
- Honors robots.txt
- Yes (documented)
- Executes JavaScript
- No — invisible to browser analytics
- Identity verifiable by IP
- No published ranges
- Official documentation
- Common Crawl docs ↗
How it identifies itself
CCBot/2.0 (https://commoncrawl.org/faq/)
User-agent strings are free to fake — any scraper can claim to be CCBot. With no published IP ranges, claims by this agent can't be cryptographically verified — treat lookalikes with suspicion.
Allow or block via robots.txt
# Block CCBot User-agent: CCBot Disallow: / # Or allow it explicitly User-agent: CCBot Allow: /
Should you block CCBot?
The broadest single lever in AI-training opt-out: one token affects every downstream model trained on Common Crawl. It also removes you from legitimate research datasets. Weigh accordingly.
Is CCBot reading your site?
CCBotnever runs JavaScript, so GA4 can't see it. VisitorType's server-side detection can — with per-page counts.
Measure it free