Cloudflare's AI bot blocking: what the one-click switch actually does (and doesn't)

Cloudflare made AI-bot blocking a checkbox: one toggle (on by default for many new sites since mid-2025) and AI crawlers get turned away at the edge before they ever reach your server. For a lot of site owners that felt like the problem solved itself.

It's the bluntest possible instrument — which is sometimes exactly right, and sometimes quietly expensive.

What the switch really does

• Blocks by category, not by consequence. The toggle doesn't distinguish training crawlers from search indexers from live assistant fetches. Blocking GPTBot is a licensing stance; blocking OAI-SearchBot de-indexes you from ChatGPT search; blocking ChatGPT-User slams the door on a live buyer's assistant. One switch, three very different outcomes.
• It enforces, unlike robots.txt. Credit where due: an edge block actually stops compliant and non-compliant crawlers it recognizes — robots.txt only ever asked nicely.
• It's invisible when it misfires. No console tells you "you've been absent from Perplexity answers for six months." The symptom is a slow flatline in AI referrals you weren't measuring anyway.

The saner sequence: measure → decide → then block

1. Measure first. Which AI agents visit, what do they read, and do any AI platforms already send you humans? Two weeks of data turns the debate from vibes into arithmetic.
2. Decide per agent, not per category. Keep search and assistant bots that feed you demand; block trainers if that's your stance — via robots.txt for the compliant, WAF rules for named offenders.
3. Keep Cloudflare for what it's uniquely good at: enforcing the blocks you chose, and rate-limiting the anonymous scrapers that never identified themselves at all.

If you run VisitorType behind Cloudflare, they compose cleanly — just don't let the edge block an agent you're trying to measure: whatever Cloudflare turns away, no analytics (ours included) will ever see.

Where VisitorType fits

VisitorType is the measure and decide layers: every AI agent identified and IP-verified, per-page reading behavior, AI referrals unmasked — so the Cloudflare toggle becomes an informed policy instead of a reflex. Create a free account, gather two weeks of evidence, then flip switches you can defend.