How to Block AI Training Bots in robots.txt

By 2026, web traffic has undergone a significant bifurcation. On one side are Search/Retrieval bots that cite your work and drive traffic; on the other are Training bots (Eaters) that ingest your data to build proprietary models without attribution or referral. Managing your robots.txt is now a matter of data sovereignty and server performance.

The Bot Taxonomy: Citers vs. Eaters

Effective blocking requires distinguishing between bot intents:

• The Eaters: Agents like GPTBot and CCBot scrape bulk data primarily for LLM training.
• The Citers: Agents like OAI-SearchBot and PerplexityBot use your content to answer user queries with links back to your site.
• The Hybrid: Googlebot remains essential for traditional search, but Google-Extended allows you to opt-out of Gemini training specifically.

Differentiating these agents allows you to maintain visibility in the "summarized internet" while restricting scrapers that offer zero ROI.

Sovereign Configuration: The Robots.txt Logic

Implement the following logic in your robots.txt file. This configuration prioritizes visibility in AI search results while cutting off the training pipeline:

# 1. Block AI Training (The Eaters)
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /


# 2. Allow Search & Citation (The Citers)
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /

Performance and Ethics

Beyond data protection, blocking aggressive crawlers preserves Server Health. Some agents, notably ClaudeBot, have been documented hitting servers with massive request volumes that do not translate to human visitors.

Benefits of a Sovereign Web strategy:

• Reduced Overhead: Blocking training bots lowers CPU usage and hosting costs.
• Preserved Logic: Prevents scraping of proprietary client-side code or unique utility logic.
• Selective Inclusion: Ensuring OAI-SearchBot is allowed prevents your site from disappearing from modern summaries.

For absolute enforcement, senior engineers should supplement this with Edge Blocking via a WAF (Web Application Firewall) to hard-block specific User-Agents at the DNS level.

Important Note: Check your logs for ClaudeBot, Bytespider, and Amazonbot to identify which "Eaters" are currently taxing your infrastructure.