How to block AI bots on WordPress — with judgment, not a kill switch

Search "block AI bots WordPress" and you'll find a hundred posts with the same robots.txt snippet blocking everything with "AI" in its name. Most of them never mention that one of those lines removes you from ChatGPT search results, and another turns away live users. Blocking is a fine choice — as three separate decisions, not one reflex.

The three decisions hiding inside "block AI"

1. Block training crawlers? (GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent…) This opts your content out of model training. It costs you no visibility anywhere. If scraping-for-training offends you, block away — this is the free one.

2. Block AI search indexers? (OAI-SearchBot, PerplexityBot…) This removes you from ChatGPT search and Perplexity results — the AI equivalent of de-indexing yourself from Google. For almost every site, don't.

3. Block assistant fetchers? (ChatGPT-User, Claude-User…) These fire when a live person asks their AI about your site. Blocking shows that person an error. Almost never worth it.

Most "block AI" tutorials implement all three and mention none. Decide each on purpose — our robots.txt generator builds the rules from exactly these three checkboxes, with current tokens for every agent.

Three ways to enforce it on WordPress

robots.txt (the polite request). WordPress serves a virtual robots.txt you can extend via your SEO plugin (Yoast/Rank Math have editors) or a robots_txt filter in your theme's functions.php. Paste the generated rules at the end — don't replace your existing directives. Reputable vendors honor this. Which brings us to…

The catch nobody mentions: robots.txt is voluntary, and the name in a user-agent string is just a claim. On our own site, roughly two-thirds of checkable "AI bot" visits were impostors — scrapers wearing GPTBot's name precisely because sites whitelist it. Your robots.txt does nothing to them, and worse, a fake GPTBot inherits whatever politeness you extended to the real one.

Server-level blocking (the actual wall). For bots that ignore robots.txt (Bytespider is the famous offender), block at the web server or CDN: an .htaccess UA condition on Apache, a firewall rule on Cloudflare. Blunt but effective — with the same caveat that UA-based rules hit the honest and the dishonest alike, and IP-based rules need the vendor's published ranges to avoid collateral damage.

Managed blocking (Cloudflare's toggle). One switch, maintained list, no site edits. The trade-off: it's the all-or-nothing reflex again — most managed toggles don't distinguish training crawlers from search indexers from assistant fetchers, so you're back to decision-by-bundle.

Verify, then re-verify quarterly

After deploying: fetch yoursite.com/robots.txt and check the rules render; then watch what actually arrives. Blocking done, the interesting question becomes who's still showing up anyway — that's your list of robots.txt-ignoring visitors, and the honest measure of what your blocks achieved. Our WordPress plugin shows exactly that per agent, verified against published IP ranges.

And revisit the decision quarterly: new agents appear monthly (the cheat sheet stays current), and the cost-benefit moves — blocking AI search in 2026 costs more visibility every quarter as those channels grow. The snippet you paste today is a policy, not a fact; treat it like one.