GPTBot, OAI-SearchBot, ChatGPT-User: Which AI Crawlers to Allow and Why
The request usually comes from legal: confirm we are not letting OpenAI train on our content. Someone adds a GPTBot disallow to robots.txt, the ticket closes, and a quarter later the CMO asks why ChatGPT recommends two competitors and never the company. Often the GPTBot line was fine. The problem is that GPTBot is one of three OpenAI agents, each feeding a different product, and robots.txt governs only two of them.
The short answer: allow OAI-SearchBot on the public pages you want eligible to surface in ChatGPT search, avoid unnecessarily blocking user-initiated ChatGPT-User fetches at the edge, and treat GPTBot as a separate business decision about model training. Blocking GPTBot does not opt a site out of ChatGPT search. Blocking OAI-SearchBot prevents that content from being surfaced in ChatGPT search answers, although OpenAI notes that navigational links may still appear in some cases. eight25, a Contentful Silver-tier Solution Partner and Contentstack Preferred Partner, checks crawler access alongside rendering, structured content, redirects, and CDN rules during enterprise CMS migration QA. Those layers matter because AI visibility depends on more than whether a bot is technically allowed through.
Three OpenAI agents, three separate decisions
Most robots.txt advice treats “AI crawlers” as one category to allow or block. OpenAI’s own documentation describes three agents with three jobs, and the consequences of blocking each one have almost nothing in common.
| Agent | What it feeds | Honors robots.txt | What blocking it costs you |
|---|---|---|---|
| GPTBot | Training data for future foundation models | Yes | Your pages stop informing future models. No effect on ChatGPT search |
| OAI-SearchBot | Crawls content used to surface websites in ChatGPT search features | Yes; OpenAI says changes can take roughly 24 hours to propagate | Your content will not be surfaced in ChatGPT search answers, although a navigational link may still appear in some cases |
| ChatGPT-User | User-initiated page fetches from ChatGPT and Custom GPTs | OpenAI says robots.txt rules may not apply because the request is user initiated | Blocking the fetch at your CDN or WAF can prevent ChatGPT from retrieving that page directly when a user asks for it |
OpenAI publishes separate IP range files for GPTBot, OAI-SearchBot, and ChatGPT-User, which gives infrastructure teams a stronger verification signal than the user-agent string alone. OpenAI also says that when a site allows both GPTBot and OAI-SearchBot, it may reuse the result of a single crawl for both purposes to avoid unnecessary duplicate crawling. The important policy point is that the two controls remain independent: a site can block GPTBot while continuing to allow OAI-SearchBot for ChatGPT search visibility.
The other assistant vendors draw the same lines with different names:
| Vendor | Training | Search index | User-initiated fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | No separate training crawler documented | PerplexityBot | Perplexity-User |
Google is the exception. Google-Extended is a product token that controls whether Googlebot-crawled content can train future Gemini models, and Google states it has no effect on Search inclusion or ranking. AI Overviews and AI Mode draw on the ordinary Googlebot crawl, so Google-Extended does not keep you out of them.
GPTBot is the only genuine judgment call
Whether GPTBot gets in is a business question, and both answers have a defensible case.
The case for blocking is legitimate. A firm that sells proprietary research, licenses its data, is negotiating content deals with AI companies, or holds a legal position on training use has good reason to keep GPTBot out. A visibility argument should not override that.
The case for allowing GPTBot is longer-term and less deterministic. OpenAI says content crawled by GPTBot may be used to improve and train future generative AI foundation models. Allowing your public product and positioning content therefore gives OpenAI the option to learn from first-party material rather than relying entirely on third-party descriptions of your company. That does not guarantee how a future model will describe the brand, but it does make first-party information available to the training pipeline. Retrieval is different: search crawlers affect whether current AI search systems can access and surface your pages when answering a question today.
For most enterprise B2B sites, the better move is to decide per content zone rather than site-wide. The file supports that directly:
# Training: allow marketing pages, keep licensed research out
User-agent: GPTBot
Allow: /
Disallow: /research/reports/
Disallow: /customer-portal/
# Search and citation: allow everything public
User-agent: OAI-SearchBot
Allow: /
Disallow: /customer-portal/
The research landing pages stay open to both agents. Openly published reports stay out of training but remain open to OAI-SearchBot, so ChatGPT search can still cite them. Whichever way the call goes, legal and marketing should make it together and write it down, because the person who edits robots.txt during a migration will not know it was a decision.
OAI-SearchBot is the one you cannot afford to block by accident
The available data suggests publishers are treating training and AI search differently. A June 2026 study examined the top 10,000 domains in the Majestic Million and successfully parsed robots.txt policies for 7,870 of them. Among those sites, 17.6% blocked GPTBot while 6.6% blocked OAI-SearchBot. Whatever the reason behind each individual policy, the aggregate pattern is clear: training crawlers are blocked much more frequently than the crawler associated with ChatGPT search. For enterprise teams, the more avoidable risk is therefore an OAI-SearchBot block introduced unintentionally through a wildcard rule, security configuration, or migration.
Accidental blocks tend to follow one of three patterns:
- A security-hardened robots.txt disallows User-agent: * and then explicitly allows Googlebot and Bingbot. Every AI crawler falls into the wildcard, including the ones that drive citations.
- Someone pastes a “block all AI bots” list from a blog post, and the list includes OAI-SearchBot, Claude-SearchBot, and PerplexityBot alongside the training crawlers.
- A preview environment’s Disallow: / ships with the new site at cutover. Someone notices the Googlebot drop in Search Console within days, while nobody is watching OAI-SearchBot.
The 24-hour propagation window OpenAI documents applies in both directions. A fix made on launch day does not restore eligibility on launch day, and a block introduced on a Friday is in force by Saturday.
ChatGPT-User takes its instructions from the user, not your robots.txt
OpenAI currently states that robots.txt rules may not apply to ChatGPT-User because these fetches are initiated by a user rather than by an automatic web crawl. In other words, ChatGPT-User should not be treated as equivalent to OAI-SearchBot or GPTBot when designing crawler policy. If an organization needs to enforce access restrictions against user-initiated fetches, that control needs to exist at the application, authentication, CDN, or WAF layer rather than relying only on robots.txt.
If those fetches genuinely need to be restricted, enforcement can happen through authentication or edge controls such as a CDN or WAF, with OpenAI’s published IP ranges providing an additional verification signal. For authenticated applications and private content that is entirely reasonable. For public B2B marketing pages, however, teams should understand the tradeoff: ChatGPT-User may be requesting the page because someone explicitly asked ChatGPT to retrieve or reason over that URL. Blocking the request can therefore prevent your first-party content from participating in that interaction.
The defensible reasons to block are narrow: authenticated content, gated assets, or a rate problem you can see in the logs. Authentication and rate limits handle those better than user-agent rules do.
Anthropic draws the line differently: its current documentation says ClaudeBot, Claude-SearchBot, and Claude-User all honor robots.txt directives. Perplexity is the cautionary case. In August 2025 Cloudflare delisted Perplexity as a verified bot, alleging undeclared crawlers that impersonated Chrome after sites blocked its declared ones. Perplexity disputed the findings. Whoever is right, the lesson holds: a user-agent string is a claim, and the IP range is the evidence.
Your CDN may have already made the decision for you
Marketing or SEO usually owns robots.txt. Security usually owns the edge. On most enterprise estates nobody has compared the two, and since 2025 the edge has been making AI crawler decisions on its own.
Cloudflare has made these decisions increasingly explicit at the infrastructure layer. In July 2025, it changed the default for new domains so AI training crawlers were blocked unless the domain owner chose otherwise. In September 2025, Cloudflare added its Content Signals Policy to managed robots.txt configurations used by more than 3.8 million domains, expressing search=yes, ai-train=no for those customers. Those signals communicate usage preferences rather than technically enforcing them, which is why Cloudflare recommends combining them with access-control and bot-management rules when enforcement matters. On September 15, 2026, Cloudflare introduced more granular Search, Training, and Agent controls. For new domains, Training and Agent traffic is blocked by default on pages displaying ads, while Search remains allowed. Cloudflare also tightened its treatment of multi-purpose crawlers: because crawlers such as Googlebot, Applebot, and BingBot can serve more than one purpose, a Training block can affect their access unless the newer purpose-specific controls are configured deliberately. Agent is the category for chat fetch bots, which is where ChatGPT-User traffic sits. The sharper edge is the multi-purpose rule: a Training block also catches any crawler that does both Search and Training, and coverage of the change names Googlebot, Applebot, and BingBot among them.
Most B2B marketing pages carry no ads, which narrows the exposure. A replatform, though, often means onboarding a new domain or zone, and the defaults land on a day nobody is reviewing crawler policy.
A quick check shows whether the edge agrees with the file. Run it from outside your corporate network:
for ua in "GPTBot/1.4" "OAI-SearchBot/1.4" "ChatGPT-User/1.0" "ClaudeBot/1.0" "Claude-SearchBot/1.0"; do
printf "%-20s " "$ua"
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; $ua)" https://www.example.com/products/
doneA 403 or a challenge page against a crawler your robots.txt allows means the two layers disagree. This test only exercises user-agent rules. Bot-management products verify crawlers by IP, so the definitive answer is in your access logs, filtered by the published IP ranges.
The AI visibility checks your CMS migration QA needs
Write the policy down before the next migration rewrites it
One of the easiest ways for an enterprise migration to lose AI visibility is to carry the wrong crawler policy into production. A staging Disallow: /, an inherited wildcard rule, or a CDN bot policy can survive cutover even when the content itself migrated correctly. On eight25’s Sitecore XP-to-Contentful migration for Qlik, roughly 5,000 pages across four regions and 19 languages were delivered in five and a half months. At that scale, crawler policy is only one item among thousands of launch decisions, which is exactly why it belongs in the migration QA process rather than being left as an informal SEO setting. On an estate the size of Qlik’s move from Sitecore XP to Contentful, roughly 5,000 pages across four regions and 19 languages delivered in 5.5 months, robots.txt is one file among thousands of decisions. That is exactly why it gets inherited unread.
The fix is to treat crawler access as a governed policy with an owner, rather than a text file with an editor. A one-page policy by content zone covers most enterprise sites:
| Content zone | Training (GPTBot, ClaudeBot) | Search (OAI-SearchBot, Claude-SearchBot) | User fetch (ChatGPT-User, Claude-User) |
|---|---|---|---|
| Product, solutions, case studies | Allow by default | Allow | Allow |
| Documentation and knowledge base | Business decision | Allow | Allow |
| Licensed research and paid reports | Block if licensing matters | Allow, at minimum the landing page | Gate with authentication |
| Customer portal and staging | Block | Block | Block via authentication |
Give the policy a named owner, review it whenever legal’s position on training changes, and make it a launch gate on every replatform. In the enterprise migration QA eight25 runs, crawler access sits alongside redirects, rendering, metadata, structured data, and content-model validation. Before DNS moves, robots.txt and edge rules are checked against the written policy; after launch, access logs confirm what the crawlers actually received. On a headless replatform, that same QA can also verify that structured content fields and relationships are making it through the frontend as crawlable, semantically clear HTML rather than disappearing behind client-side rendering or presentation logic.
Frequently Asked Questions
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot crawls content that may be used to improve and train future OpenAI generative AI models. OAI-SearchBot is managed separately and is used to surface websites in ChatGPT search. OpenAI explicitly supports allowing OAI-SearchBot while disallowing GPTBot, so an organization can opt out of training access without opting out of ChatGPT search visibility.
Should a B2B company block AI crawlers from training on its content?
It depends on what the content is worth outside your site. If you sell research, license data, or hold a legal position on training use, blocking training crawlers such as GPTBot and ClaudeBot is defensible. If the pages are product and positioning content, allowing training means future models learn your category from your own description rather than from competitors and review sites.
Why is ChatGPT-User still visiting pages my robots.txt disallows?
OpenAI states that robots.txt rules may not apply to ChatGPT-User because those fetches are initiated by a person using ChatGPT or a Custom GPT. Organizations that need stronger enforcement should use controls such as authentication, application rules, or CDN/WAF policies rather than relying on robots.txt alone. OpenAI publishes ChatGPT-User IP ranges that can help infrastructure teams verify legitimate traffic. For public marketing content, teams should weigh any block carefully because it can prevent ChatGPT from retrieving the first-party page when a user explicitly asks for it.
Does Google-Extended stop my content appearing in AI Overviews?
No. Google-Extended controls whether Google-crawled content can be used for certain Gemini training and grounding uses, and Google says it does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search and rely on the normal Googlebot crawl. To limit what can appear from a page in those experiences, Google points site owners to standard Search controls such as nosnippet, data-nosnippet, max-snippet, and noindex.
Who can help audit AI crawler access during a CMS migration?
Look for a migration partner that treats AI visibility as an architecture and QA problem rather than only an SEO setting. That means checking robots.txt, CDN/WAF rules, crawler responses, rendering, structured data, metadata, redirects, and the underlying content model together. eight25 applies those checks during enterprise replatforms and then validates crawler access again from production logs after cutover. For headless CMS migrations in particular, this approach also helps confirm that structured content created in the CMS is reaching the rendered page in a form search engines and AI retrieval systems can actually parse.
Before your next robots.txt edit, find out what your edge is already doing
If your team is about to change crawler policy or replatform, start by comparing what robots.txt says, what the CDN actually enforces, and what the access logs show each crawler received. Then go one layer deeper: check whether the content those crawlers reach is structured, rendered, and connected clearly enough for machines to interpret it correctly. eight25 builds both sides of that problem into enterprise CMS migration discovery and QA, particularly on headless platforms such as Contentful and Contentstack, so a replatform improves not only how teams manage content but also how reliably that content can be discovered and understood by the next generation of search and AI interfaces.