GPTBot, ClaudeBot, Google-Extended: a robots.txt guide to AI crawlers for people who run websites

5.1First, what robots.txt is and is not

The Robots Exclusion Protocol was published by the IETF in 2022 as RFC 9309, a Proposed Standard. Two lines from it are worth keeping in mind:

• “These rules are not a form of access authorization.” robots.txt is a request that well-behaved crawlers honour. It does not lock anything. If content must stay private, put it behind authentication.
• Crawlers “SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable.” Changes are not instant.

The RFC also says “the most specific match found MUST be used” when rules conflict, and that on a 5xx error for the robots.txt file itself, “the crawler MUST assume complete disallow.” A broken robots.txt can therefore block well-behaved crawlers entirely.

5.2Google: Google-Extended is not Googlebot

Google’s crawler documentation describes Google-Extended as “a standalone product token that web publishers can use to manage whether content Google crawls from their sites” may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI.

Two sentences matter most:

• “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”
• “Google-Extended doesn’t have a separate HTTP request user agent string.” It is a control token, not a crawler you will see in your logs. Crawling is still done by Google’s normal crawlers.

So:

User-agent: Google-Extended
Disallow: /

opts your content out of those Gemini uses, and leaves Google Search alone. It does not remove you from AI Overviews in Search. Those are governed by Search’s own snippet controls, which is a separate topic.

5.3OpenAI: three agents, three decisions

OpenAI’s crawler overview lists three user agents, and states that “each setting is independent of the others.”

• GPTBot “is used to make our generative AI foundation models more useful and safe.” OpenAI says “disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”
• OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features.” OpenAI is explicit about the trade-off: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers,” though they can still appear as navigational links.
• ChatGPT-User is used “when users ask ChatGPT or a CustomGPT a question” and it visits a page on their behalf. OpenAI notes: “Because these actions are initiated by a user, robots.txt rules may not apply.”

It also says it “can take ~24 hours from a site’s robots.txt update for our systems to adjust.”

5.4Anthropic: the same three-way split

Anthropic’s help article describes three bots:

• ClaudeBot collects web content that could contribute to model training. Blocking it signals that “the site’s future materials should be excluded from our AI model training datasets.”
• Claude-User is used “when individuals ask questions to Claude.” Blocking it “may reduce your site’s visibility for user-directed web search.”
• Claude-SearchBot “navigates the web to improve search result quality for users.” Blocking it “prevents our system from indexing your content for search optimization.”

Anthropic says its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt,” and that it supports “the non-standard Crawl-delay extension.” One practical note from the same page: “Please do this for every subdomain that you wish to opt out from.” Each subdomain serves its own robots.txt.

5.5Common Crawl: the dataset many others use

Common Crawl’s CCBot builds an open web archive. Its page gives the user agent as CCBot/2.0 and the block as:

User-agent: CCBot
Disallow: /

It warns that it is “aware of crawlers falsely identifying themselves as CCBot,” and publishes its IP ranges so you can verify real traffic. That warning applies to every bot on this page: a user-agent string is easy to fake.

5.6Three policies, written out

Here are three complete policies for different goals. Pick one; do not combine them blindly.

1. Stay in AI search and assistants, opt out of training.

User-agent: GPTBot
Disallow: /
 
User-agent: ClaudeBot
Disallow: /
 
User-agent: Google-Extended
Disallow: /
 
User-agent: CCBot
Disallow: /
 
User-agent: *
Allow: /

OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User fall through to the * group and stay allowed. For most small businesses that want to be found, this is the setting I would start from.

2. Allow everything. An empty Disallow: under User-agent: *, or no AI-specific lines at all. Reasonable for a business whose goal is maximum visibility.

3. Opt out of every documented AI agent. Add OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User to the blocked list. Understand the cost: by the vendors’ own descriptions, you will not be shown in ChatGPT search answers, and you may lose visibility in Claude’s user-directed search.

5.7The mistakes I see most

• Blocking Google-Extended to “get out of AI Overviews.” Google says it does not affect Search.
• Blocking only the training bot and assuming you are invisible to AI. Search and user-triggered agents are separate tokens.
• Blocking the search bots by accident with a broad User-agent: * / Disallow: / meant for something else.
• Forgetting subdomains. shop.example.com needs its own file.
• Treating robots.txt as security. The RFC says it is not.

Write down which policy you chose and why, in a comment at the top of the file. The next person to edit it, possibly you in a year, will need to know. I’m Sal Talash, and I write about how websites and AI systems fit together at saltalash.vercel.app.