AI Crawler Cheat Sheet
Every major AI agent on one page: how it identifies itself, the token that controls it in robots.txt, whether it actually honors robots.txt, and whether its IPs can be verified against published ranges. Generated from the same data as our agent directory and detection engine, so it stays current as agents change.
AI training crawlers
| Agent | Vendor | robots.txt token | Honors robots.txt | IP-verifiable | Docs |
|---|---|---|---|---|---|
| GPTBot OpenAI's training crawler — feeds future GPT models. | OpenAI | GPTBot | yes | yes — ranges published | official |
| ClaudeBot Anthropic's training crawler for Claude models. | Anthropic | ClaudeBot | yes | no | official |
| CCBot The nonprofit crawl behind many AI training datasets. | Common Crawl | CCBot | yes | no | official |
| Google-Extended The robots.txt switch for Gemini training — not a crawler you'll see in logs. | Google-Extended | yes | no | official | |
| GoogleOther Google's generic crawler for R&D and non-Search products. | GoogleOther | yes | yes — ranges published | official | |
| Meta-ExternalAgent Meta's crawler for AI training and product indexing. | Meta | meta-externalagent | yes | no | official |
| Bytespider ByteDance's aggressive, sparsely documented crawler. | ByteDance | Bytespider | unclear | no | — |
AI search crawlers
| Agent | Vendor | robots.txt token | Honors robots.txt | IP-verifiable | Docs |
|---|---|---|---|---|---|
| OAI-SearchBot The crawler that builds ChatGPT search's index. | OpenAI | OAI-SearchBot | yes | yes — ranges published | official |
| PerplexityBot Perplexity's index crawler — the source of its citations. | Perplexity | PerplexityBot | yes | yes — ranges published | official |
AI assistant (user-triggered)s
| Agent | Vendor | robots.txt token | Honors robots.txt | IP-verifiable | Docs |
|---|---|---|---|---|---|
| ChatGPT-User A live ChatGPT user's assistant, fetching your page mid-conversation. | OpenAI | ChatGPT-User | yes | yes — ranges published | official |
| Claude-User Claude fetching a page for the person in the conversation. | Anthropic | Claude-User | yes | no | official |
| Perplexity-User Perplexity fetching a page because a user asked about it. | Perplexity | Perplexity-User | unclear | yes — ranges published | official |
“IP-verifiable” means the vendor publishes official IP ranges — paste any visit into the free AI Bot Checker to test a claim. Data maintained with daily range refreshes; last generated at build time.
This table rots — new agents appear monthly.
Get an email when a new AI agent starts crawling the web or a vendor changes its ranges. No newsletter filler — just the changes. Unsubscribe anytime.
By subscribing you consent to receive these update emails. Unsubscribe link in every one.