robots.txt for AI crawlers: allow or block GPTBot, ClaudeBot and PerplexityBot

AI companies run several different bots. Some collect pages for model training, some build an AI search index, and some fetch one page because a user asked. You can treat each group differently in robots.txt. This page lists the user-agent names, gives three tested setups you can copy, and shows how to check what your current file really allows.

Opens the free Agent-Ready Scan with your URL. No account. 10 scans per IP address per day. It checks robots.txt for 12 AI crawlers plus llms.txt, sitemap, schema.org, no-JavaScript rendering and agent/API discovery files.

The short answer

AI crawler user-agents and what they do

User-agent tokenOperatorTypeWhat it does
GPTBotOpenAITrainingCollects pages that may be used to train OpenAI models.
OAI-SearchBotOpenAISearchIndexes pages so they can appear in ChatGPT search answers.
ChatGPT-UserOpenAIUser-triggeredFetches a page when a ChatGPT user (or a custom GPT action) asks for it.
ClaudeBotAnthropicTrainingCollects pages that may be used to train Claude models.
Claude-SearchBotAnthropicSearchIndexes pages to improve Claude's search results.
Claude-UserAnthropicUser-triggeredFetches a page when a Claude user asks a question that needs it.
PerplexityBotPerplexitySearchIndexes pages for Perplexity search answers.
Perplexity-UserPerplexityUser-triggeredFetches a page when a Perplexity user asks for it.
Google-ExtendedGoogleTrainingNot a separate crawler: a robots.txt token. Disallowing it tells Google not to use pages Googlebot already crawled for Gemini model training and grounding. It does not affect Google Search.
Applebot-ExtendedAppleTrainingAlso a token, not a crawler: controls whether Applebot-crawled pages may be used to train Apple's models.
CCBotCommon CrawlTrainingBuilds the public Common Crawl dataset, which many AI labs use for training.
BytespiderByteDanceTrainingByteDance's crawler, widely reported as collecting data for its AI models.
meta-externalagentMetaTrainingCollects pages for Meta's AI products, including model training.

Operators add and rename bots from time to time; check each company's crawler documentation before relying on a list. User-triggered fetchers act on behalf of a person, and operators do not all treat robots.txt the same way for them.

Three robots.txt setups you can copy

1. Allow all crawlers, including AI

# Allow everyone, including AI crawlers
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

With no rule naming an AI bot, every crawler falls back to the * group. This is also what happens when you have no robots.txt at all.

2. Visible in AI search, opted out of training

# Stay visible in AI search and answers, opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

# Everyone else (Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User,
# Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User ...)
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Several User-agent lines in a row share the rules that follow them, so one Disallow: / covers all seven training bots.

3. Block AI crawlers, keep normal search engines

# Block known AI crawlers and AI-use tokens, keep normal search engines
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Googlebot and Bingbot are not affected. Blocking Google-Extended does not change your Google Search ranking or inclusion.

Mistakes we see most

Real examples (scanned 22 November 2026)

We ran the free scan on four well-known homepages. These are their own choices, not mistakes:

Siterobots.txt result for the 12 AI crawlers the scan tests
nytimes.comAll 12 blocked (each has its own Disallow: / group).
theguardian.com8 blocked: ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Applebot-Extended, CCBot, Bytespider, meta-externalagent. The OpenAI bots and Google-Extended are not named, so they fall back to the * group and are allowed.
github.comOnly Bytespider blocked.
stripe.comAll 12 allowed, and the site also publishes a 69,770-character llms.txt.

The scan's robots.txt result for github.com, as returned by the API:

{
 "id": "robots_ai",
 "name": "robots.txt lets AI agents in",
 "points": 5,
 "max": 10,
 "status": "partial",
 "detail": "Blocked for this page: Bytespider.",
 "fix": "If you want to be read and cited by AI assistants, allow their crawlers (for example OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) in robots.txt. Blocking training bots only is a valid choice.",
 "blocked": [
  "Bytespider"
 ]
}

Check your own site

Opens the free Agent-Ready Scan with your URL. No account. 10 scans per IP address per day. It checks robots.txt for 12 AI crawlers plus llms.txt, sitemap, schema.org, no-JavaScript rendering and agent/API discovery files.

Or call the API directly (free, no key, same 10-per-day limit):

curl "https://manhourplan.com/agent-ready/api/scan?url=https://your-site.com/"

The response is JSON with a 0–100 score, a grade, and one result per check. Look for "id": "robots_ai": blocked lists the AI crawlers your robots.txt shuts out for that page. The scan reads robots.txt as the standard describes; it does not see CDN or firewall blocks, and it cannot tell you whether an AI assistant will cite your site.

Want it set up for you?

If you would rather not edit robots.txt, llms.txt, sitemap and structured data yourself, our Agent-Ready Website Setup does it for a fixed $499 one-time fee: we set your robots.txt rules for AI crawlers the way you choose (for example AI search yes, training no), write and publish llms.txt, check your sitemap, add schema.org JSON-LD to up to 10 key pages, and send a before-and-after scan report. We aim to finish within 5 business days after we have access. It does not include any promise of AI citations or rankings. Details, scope and checkout are on the Agent-Ready page.

Building an AI agent that reads websites instead? PageMD turns a URL into clean Markdown for agents, and SiteDoctor audits pages over MCP.