Bytespider
AI training crawlerByteDance's aggressive, sparsely documented crawler.
What Bytespider does
Bytespider crawls for ByteDance (TikTok's parent), widely assumed to feed its AI models. It has a reputation for high request volume, no published IP ranges, and historically inconsistent robots.txt behavior — the least polite guest in this directory.
Fact sheet
- Operator
- ByteDance
- robots.txt token
- Bytespider
- Honors robots.txt
- Unclear / inconsistent
- Executes JavaScript
- No — invisible to browser analytics
- Identity verifiable by IP
- No published ranges
- Official documentation
- None published
How it identifies itself
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])
User-agent strings are free to fake — any scraper can claim to be Bytespider. With no published IP ranges, claims by this agent can't be cryptographically verified — treat lookalikes with suspicion.
Allow or block via robots.txt
# Block Bytespider User-agent: Bytespider Disallow: / # Or allow it explicitly User-agent: Bytespider Allow: /
Should you block Bytespider?
With no published ranges to verify and shaky robots.txt compliance, many operators block it at the edge (WAF/user-agent rule) rather than trusting robots.txt. There is no documented downside for Western sites: it powers no search or assistant channel that sends you traffic.
Is Bytespider reading your site?
Bytespidernever runs JavaScript, so GA4 can't see it. VisitorType's server-side detection can — with per-page counts.
Measure it free