robots.txt for AI crawlers: the complete guide (with every token that matters)

Every "block AI bots" tutorial shows you the same three lines of robots.txt. Almost none of them tell you what each line costs — or that robots.txt is a polite request some bots simply ignore.

The tokens, and what blocking each one does

Token	Vendor	Blocking it means
GPTBot	OpenAI	Your content stops feeding model training. ChatGPT search can still cite you
OAI-SearchBot	OpenAI	You disappear from ChatGPT search — AI de-indexing
ChatGPT-User	OpenAI	A live user's assistant gets an error instead of your page
ClaudeBot	Anthropic	Same trade as GPTBot, for Claude
PerplexityBot	Perplexity	Gone from Perplexity answers
Google-Extended	Google	Opts out of Gemini training — normal Google Search is untouched
CCBot	Common Crawl	Out of the dataset many smaller models train on
Amazonbot / Meta-ExternalAgent / Applebot-Extended	Amazon / Meta / Apple	Their assistants and training pipelines

The pattern: training tokens are a philosophy question; search and assistant tokens are distribution. Blocking OAI-SearchBot because you're angry at GPTBot is de-indexing yourself from the fastest-growing search surface out of spite.

A sane starting point for most sites:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

…and leave search/assistant tokens alone until your own data says otherwise.

The part nobody says out loud

robots.txt is voluntary. Compliant vendors honor it; scrapers wearing a fake GPTBot costume never did. So a robots.txt change should be verified, not assumed: did visits from that agent actually stop, and is whatever still arriving under that name coming from the vendor's real IP ranges?

Our AI agent directory lists every agent's token, whether it respects robots.txt, and whether its identity is IP-verifiable — check it before you copy a block list from a 2023 blog post.

Where VisitorType fits

VisitorType is the before-and-after picture for every robots.txt decision: which AI agents visit, what they read, whether they honored your block — and whether the "GPTBot" that ignored it was OpenAI at all (we verify every claimed crawler against vendor-published IP ranges, and downgrade the impostors). Decide with a month of data, not a template: create a free account and look before you block.