The question “Should I close my site to AI bots?” has no single answer, because “AI bot” is not one thing. The same company has several crawlers working for different purposes, and each can be managed separately in robots.txt. Let's clarify the distinction first.
Three types of bots
- Training bots: collect content to develop generative models.
- Search / citation bots: crawl to show and link your site in AI-powered search features.
- User-triggered bots: run when a user asks an AI assistant for a page.
Bot table
The classification below is based on the companies' own official documentation.
| Company | Bot | Type | Note |
|---|---|---|---|
| OpenAI | GPTBot | Training | Blocking asks that content not be used in training |
| OpenAI | OAI-SearchBot | Search / citation | Blocking removes the site from ChatGPT search answers |
| OpenAI | ChatGPT-User | User-triggered | Because it is user-initiated, robots.txt rules may not apply |
| Anthropic | ClaudeBot | Training | Blocked with Disallow |
| Anthropic | Claude-SearchBot | Search | Visibility may decrease if blocked |
| Anthropic | Claude-User | User-triggered | Accesses the site when a user asks |
| Perplexity | PerplexityBot | Search / citation | To show sites in Perplexity results; stated not to be used for foundation model training |
| Perplexity | Perplexity-User | User-triggered | Generally does not follow robots.txt rules |
| Apple | Applebot-Extended | Training control | Does not crawl; determines whether the data may be used to train Apple's generative models |
| Common Crawl | CCBot | Open data archive | Open web archive of a nonprofit foundation |
OpenAI states that a robots.txt change can take about 24 hours to be reflected in its systems.
Google: Googlebot and Google-Extended
Google has two separate concepts:
- Googlebot is Google Search's main crawler. Access control for AI Overviews and AI Mode also depends on Googlebot's robots.txt rules and on preview controls (
nosnippet,max-snippet,noindex). If you block Googlebot, you also disappear from search. - Google-Extended is not a separate crawler, only a token used in robots.txt. It governs the training of Gemini Apps and Vertex AI Gemini models and the use of “Grounding with Google Search”. According to Google, it does not affect inclusion in Google Search and is not a ranking signal.
What does robots.txt not do?
robots.txt is a voluntary protocol defined in the IETF's RFC 9309; it is not access authorization or a security mechanism. For crawlers that comply, it is a request; it does not technically block a client that does not comply. You should protect truly confidential content with authentication, not with robots.txt.
Decision framework
| Your goal | Approach |
|---|---|
| Appear in AI searches | Allow search/citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) |
| You don't want your content used in model training | Block training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended); if you wish, block the open web archive CCBot too. You can keep allowing search bots |
| Both visibility and full control | Decide what you want first; write a deliberate line for each bot |
An example robots.txt for a visibility-focused corporate site can be seen in this site's robots.txt file. This is a business decision: allowing training bots means your content may be used to train models.
Frequently asked questions
If I block all AI bots, will I disappear from Google too?
You disappear from Google search only if you block Googlebot. Blocking Google-Extended does not affect inclusion in Google Search.
Why might user-triggered bots not follow robots.txt?
According to the providers' documentation, these bots act on a user's direct request, so robots.txt rules may not apply.
Sources
- OpenAI: Bots
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google: Common crawlers
- Google: AI features and your website
- Apple: About Applebot
- Common Crawl: CCBot
- IETF: RFC 9309 Robots Exclusion Protocol
Sources checked on 1 October 2026; official pages may change over time.
