AI companies run several different bots. Some collect pages for model training, some build an AI search index, and some fetch one page because a user asked. You can treat each group differently in robots.txt. This page lists the user-agent names, gives three tested setups you can copy, and shows how to check what your current file really allows.
Opens the free Agent-Ready Scan with your URL. No account. 10 scans per IP address per day. It checks robots.txt for 12 AI crawlers plus llms.txt, sitemap, schema.org, no-JavaScript rendering and agent/API discovery files.
OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User).GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent). That is a valid choice and does not remove you from Google Search.| User-agent token | Operator | Type | What it does |
|---|---|---|---|
GPTBot | OpenAI | Training | Collects pages that may be used to train OpenAI models. |
OAI-SearchBot | OpenAI | Search | Indexes pages so they can appear in ChatGPT search answers. |
ChatGPT-User | OpenAI | User-triggered | Fetches a page when a ChatGPT user (or a custom GPT action) asks for it. |
ClaudeBot | Anthropic | Training | Collects pages that may be used to train Claude models. |
Claude-SearchBot | Anthropic | Search | Indexes pages to improve Claude's search results. |
Claude-User | Anthropic | User-triggered | Fetches a page when a Claude user asks a question that needs it. |
PerplexityBot | Perplexity | Search | Indexes pages for Perplexity search answers. |
Perplexity-User | Perplexity | User-triggered | Fetches a page when a Perplexity user asks for it. |
Google-Extended | Training | Not a separate crawler: a robots.txt token. Disallowing it tells Google not to use pages Googlebot already crawled for Gemini model training and grounding. It does not affect Google Search. | |
Applebot-Extended | Apple | Training | Also a token, not a crawler: controls whether Applebot-crawled pages may be used to train Apple's models. |
CCBot | Common Crawl | Training | Builds the public Common Crawl dataset, which many AI labs use for training. |
Bytespider | ByteDance | Training | ByteDance's crawler, widely reported as collecting data for its AI models. |
meta-externalagent | Meta | Training | Collects pages for Meta's AI products, including model training. |
Operators add and rename bots from time to time; check each company's crawler documentation before relying on a list. User-triggered fetchers act on behalf of a person, and operators do not all treat robots.txt the same way for them.
# Allow everyone, including AI crawlers User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
With no rule naming an AI bot, every crawler falls back to the * group. This is also what happens when you have no robots.txt at all.
# Stay visible in AI search and answers, opt out of model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent Disallow: / # Everyone else (Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, # Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User ...) User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Several User-agent lines in a row share the rules that follow them, so one Disallow: / covers all seven training bots.
# Block known AI crawlers and AI-use tokens, keep normal search engines User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Googlebot and Bingbot are not affected. Blocking Google-Extended does not change your Google Search ranking or inclusion.
* group. Under the robots.txt standard (RFC 9309) a crawler obeys only the most specific group that names it and ignores User-agent: *. So if you add a group for GPTBot, copy into it every Disallow line it should still obey:User-agent: * Disallow: /admin/ # Looks harmless, but GPTBot now ignores the * group above # and is allowed into /admin/: User-agent: GPTBot Allow: /Our own robots.txt repeats
Disallow: /credits/checkout in every named group for this reason.www.example.com/robots.txt does not cover example.com, docs.example.com or shop.example.com.Sitemap: https://your-site/sitemap.xml so crawlers find every page.We ran the free scan on four well-known homepages. These are their own choices, not mistakes:
| Site | robots.txt result for the 12 AI crawlers the scan tests |
|---|---|
| nytimes.com | All 12 blocked (each has its own Disallow: / group). |
| theguardian.com | 8 blocked: ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Applebot-Extended, CCBot, Bytespider, meta-externalagent. The OpenAI bots and Google-Extended are not named, so they fall back to the * group and are allowed. |
| github.com | Only Bytespider blocked. |
| stripe.com | All 12 allowed, and the site also publishes a 69,770-character llms.txt. |
The scan's robots.txt result for github.com, as returned by the API:
{
"id": "robots_ai",
"name": "robots.txt lets AI agents in",
"points": 5,
"max": 10,
"status": "partial",
"detail": "Blocked for this page: Bytespider.",
"fix": "If you want to be read and cited by AI assistants, allow their crawlers (for example OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) in robots.txt. Blocking training bots only is a valid choice.",
"blocked": [
"Bytespider"
]
}
Opens the free Agent-Ready Scan with your URL. No account. 10 scans per IP address per day. It checks robots.txt for 12 AI crawlers plus llms.txt, sitemap, schema.org, no-JavaScript rendering and agent/API discovery files.
Or call the API directly (free, no key, same 10-per-day limit):
curl "https://manhourplan.com/agent-ready/api/scan?url=https://your-site.com/"
The response is JSON with a 0–100 score, a grade, and one result per check. Look for "id": "robots_ai": blocked lists the AI crawlers your robots.txt shuts out for that page. The scan reads robots.txt as the standard describes; it does not see CDN or firewall blocks, and it cannot tell you whether an AI assistant will cite your site.
If you would rather not edit robots.txt, llms.txt, sitemap and structured data yourself, our Agent-Ready Website Setup does it for a fixed $499 one-time fee: we set your robots.txt rules for AI crawlers the way you choose (for example AI search yes, training no), write and publish llms.txt, check your sitemap, add schema.org JSON-LD to up to 10 key pages, and send a before-and-after scan report. We aim to finish within 5 business days after we have access. It does not include any promise of AI citations or rankings. Details, scope and checkout are on the Agent-Ready page.
Building an AI agent that reads websites instead? PageMD turns a URL into clean Markdown for agents, and SiteDoctor audits pages over MCP.