AnswerReadyAI crawlers: the full list and how to set robots.txt
Updated
AI companies run two kinds of crawler: ones that fetch pages to answer a question right now, and ones that collect training data. Blocking the first kind removes you from AI answers; blocking the second is a business choice.
Which AI crawlers should I allow?
Allow the crawlers that fetch pages for AI search and user requests: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User, plus Googlebot and Bingbot, which feed Google AI Overviews and Microsoft Copilot. Blocking these keeps you out of their answers.
What are the main AI crawler user agents?
These are the user-agent tokens published by each company. Check their official documentation before relying on any one of them, as they change.
| Company | User agent | Purpose | Affects AI answers? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Indexes pages for ChatGPT search | Yes |
| OpenAI | ChatGPT-User | Fetches a page when a ChatGPT user asks | Yes |
| OpenAI | GPTBot | Collects model training data | No (training only) |
| Anthropic | Claude-SearchBot | Indexes pages for Claude search | Yes |
| Anthropic | Claude-User | Fetches a page when a Claude user asks | Yes |
| Anthropic | ClaudeBot | Collects model training data | No (training only) |
| Perplexity | PerplexityBot | Indexes pages for Perplexity answers | Yes |
| Perplexity | Perplexity-User | Fetches a page when a user asks | Yes |
| Googlebot | Google Search, including AI Overviews and AI Mode | Yes | |
| Google-Extended | robots.txt token controlling Gemini training use | No (not used for Search) | |
| Microsoft | Bingbot | Bing search, which powers Copilot answers | Yes |
| Apple | Applebot / Applebot-Extended | Siri and Spotlight / Apple AI training opt-out | Applebot yes; Extended no |
| Common Crawl | CCBot | Open web archive used by many AI labs | No (training) |
| Meta | Meta-ExternalAgent | AI training and products | Mostly training |
How do I allow AI search but block AI training in robots.txt?
Name the training crawlers and disallow them, and leave everything else allowed. This keeps you in AI search answers while opting out of model training.
# Allow everyone by default (includes AI search crawlers)
User-agent: *
Allow: /
# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlWhy is my site blocking AI crawlers when robots.txt allows them?
A firewall or CDN setting can block them before robots.txt is ever read. Cloudflare, for example, offers a one-click "block AI bots" setting, and security plugins such as Wordfence can block unfamiliar user agents. Check those settings, then run an audit to confirm what an AI crawler actually receives.
Do AI crawlers run JavaScript?
Most don't. If your text only appears after JavaScript runs, many AI crawlers see an almost empty page. Server-side rendering or pre-rendering fixes this.
Check your own site
The free AEO audit runs 25 checks in about 30 seconds and shows exactly what to fix first.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages on request, so blocking only GPTBot keeps you eligible for ChatGPT search answers.
Does Google-Extended affect AI Overviews?
No. Google says Google-Extended controls use of content for Gemini model training and grounding, not Google Search. AI Overviews use the normal Googlebot crawl.
How can I test whether AI crawlers can reach my site?
Run our free AEO audit. It reads your robots.txt for each AI crawler and requests your homepage the way an AI assistant does, so it can spot firewall blocks too.
Related guides
llms.txt guide
llms.txt is a proposed Markdown file that summarises your website for AI tools. What it is, a copy-paste template, and an honest look at whether AI companies use it.
AEO checklist
A practical answer engine optimization checklist: crawler access, rendering, structured data, answer-shaped content, entity details and freshness. The same 25 checks our free audit runs.
Get recommended by ChatGPT
How ChatGPT search finds and chooses businesses to recommend, and the steps that make yours easier to cite: crawler access, clear answers, structured data and reviews.