AI Agent Directory
The AI crawlers, search bots, and assistant fetchers reading the web right now — what each one does, how it identifies itself, and whether blocking it costs you anything. Facts match the detection engine that powers VisitorType.
AI training crawlers
OpenAI's training crawler — feeds future GPT models.
Anthropic's training crawler for Claude models.
The nonprofit crawl behind many AI training datasets.
The robots.txt switch for Gemini training — not a crawler you'll see in logs.
Google's generic crawler for R&D and non-Search products.
Meta's crawler for AI training and product indexing.
ByteDance's aggressive, sparsely documented crawler.
AI search crawlers
AI assistant (user-triggered)s
A live ChatGPT user's assistant, fetching your page mid-conversation.
Claude fetching a page for the person in the conversation.
Perplexity fetching a page because a user asked about it.
Which of these are on your site right now?
One snippet + one plugin shows every AI agent reading your pages — verified against vendor IP ranges, not just user-agent strings.
See your AI visitors — free