robots.txt for AI crawlers
robots.txt tells well-behaved crawlers which parts of your site they may fetch. With AI products, the key decision is separating crawlers that find pages for answers from crawlers that collect training data.
The main crawlers
| User agent | Operator | Purpose |
|---|---|---|
| Googlebot | Search indexing, also used by AI Overviews and AI Mode | |
| Bingbot | Microsoft | Bing indexing, feeds Copilot |
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks for it |
| GPTBot | OpenAI | Collects content that may train models |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers |
| ClaudeBot | Anthropic | Collects content that may train models |
| Claude-SearchBot | Anthropic | Indexes pages for Claude search results |
| Google-Extended | Opt-out token for Gemini training and grounding. Does not affect Search ranking | |
| Applebot-Extended | Apple | Opt-out token for Apple AI training |
| CCBot | Common Crawl | Open web dataset used by many AI projects |
Operators change names and add bots, so confirm against each company's current documentation.
Example: allow search and answers, block training
User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / Sitemap: https://www.example.com/sitemap.xml
This keeps Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot unrestricted, because they fall under the * group.
Important caveats
- robots.txt is voluntary. Reputable crawlers follow it. It is not security and does not hide private pages.
- A bot-specific group replaces the
*group for that bot. Rules are not merged. - Blocking Googlebot removes you from Google Search. Blocking Google-Extended does not.
- Blocking does not delete old data. Content already collected may remain.
- Changes take time. Crawlers re-read robots.txt periodically.
Test your file
Paste your site into the robots.txt and sitemap checker. It shows, for each crawler above, whether your root is allowed or blocked, plus your sitemap and llms.txt status.
Frequently asked questions
Should I block AI crawlers?
Blocking search-oriented bots can keep you out of AI answers. Blocking training-only bots is a content-rights choice that does not stop citation. Decide based on your goals.
Where does robots.txt go?
At the root of the host, for example https://example.com/robots.txt. Each subdomain needs its own.
Related
Want the issues fixed for you?
RankFlow Systems handles the fixes and the content.