robots.txt generator for AI crawlers

“Block AI bots” is three separate decisions, and most robots.txt guides conflate them. Blocking training crawlers is a licensing choice with no traffic cost. Blocking AI search de-lists you from ChatGPT and Perplexity results. Blocking assistant fetches turns away a live user whose AI is reading your page for them. Pick each one deliberately:

Block AI training crawlersGPTBot, ClaudeBot, Meta-ExternalAgent, CCBot… Opts your content out of model training. No visibility cost.Block AI search indexersOAI-SearchBot, PerplexityBot… ⚠️ Removes you from ChatGPT/Perplexity search results and citations — the AI equivalent of de-indexing from Google.Block assistant fetchesChatGPT-User, Claude-User… ⚠️ These fire when a live person asks their AI about your site. Blocking shows that person an error.

Your robots.txt rules

# AI-crawler rules — generated with visitortype.ai/tools/robots-txt-generator
# Paste at the END of your existing robots.txt (keep your current rules).

# Nothing blocked — all AI agents may read your site (robots.txt unchanged).

# Notes:
# - robots.txt is voluntary; reputable vendors honor it, copycats don't.
# - Bytespider (ByteDance) is widely reported to ignore robots.txt.
# - Blocking here does NOT remove you from classic Google/Bing search.

robots.txt is a request, not a lock.

On our own site, ~2/3 of checkable “AI bot” visits were impostors ignoring it entirely. VisitorType shows you who actually reads your pages — verified by IP.

See who really visits — free