Blocking “the OpenAI bot” is not a policy. It is an unresolved choice hidden inside a singular noun.
Three OpenAI user-agents sit behind that noun, and each answers a different question. GPTBot crawls content that may be used to train OpenAI’s foundation models; disallowing it tells OpenAI your content should not be used for that training. OAI-SearchBot decides whether your pages can surface in ChatGPT search answers. ChatGPT-User fetches a page because a person asked ChatGPT or a Custom GPT to look at it. The names rhyme. The controls do not.
If you want ChatGPT Search to find your public pages while keeping them out of future training, you need two separate instructions. If you want the reverse, write the reverse pair. A rule for ChatGPT-User settles neither question.
The comparison
| User-agent | What starts the visit | Published purpose | What a robots.txt disallow does |
|---|---|---|---|
GPTBot | Automatic crawling | Content that may be used to make OpenAI’s generative AI foundation models more useful and safe | Signals that your content should not be used in training those models |
OAI-SearchBot | Automatic crawling for Search | Surfacing websites in ChatGPT’s search features | Keeps your content out of ChatGPT search answers; a navigational link may still appear |
ChatGPT-User | A person’s request in ChatGPT or a Custom GPT | Visiting a page to fulfil that request | Little, on OpenAI’s own account: robots.txt rules may not apply, and this agent plays no part in Search inclusion |
OpenAI says the GPTBot and OAI-SearchBot settings are independent. It also says that when both are allowed, it may use one crawl for both purposes. Shared fetching does not merge the permissions. You still decide twice.
The same crawler page lists a fourth agent, OAI-AdsBot, which validates landing pages submitted as ChatGPT ads. That is a separate job, and it stays outside this comparison.
Two robots.txt policies
Search on, training off:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Training on, Search off:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Disallow: /Neither block is safe to paste blind. Under the robots.txt standard, a crawler follows the group that names it most specifically and falls back to User-agent: * only when no group does. A path rule, a second matching group, a CDN bot setting, or a firewall rule can change the effective answer. Test representative URLs against the whole file, not against the stanza you added. OpenAI publishes IP ranges for each agent. Use them when you need to tell a declared user-agent from an impersonator.
There is no Search opt-out hidden inside ChatGPT-User. OpenAI assigns that job to OAI-SearchBot. If a page must stay unreachable even when a person asks a tool to fetch it, put it behind access control. robots.txt is an instruction to crawlers, not a lock.
What each change does not do
Disallowing GPTBot does not remove a page from ChatGPT Search. It also does not establish that anything fetched before the change has left a trained model. The control is a forward signal about training use. It is not a retroactive eraser.
Disallowing OAI-SearchBot keeps your content out of ChatGPT search answers, on OpenAI’s stated policy. A title and a link can still surface as navigation. If that residue matters to you, read the current publisher guidance before treating robots.txt as a removal mechanism.
A ChatGPT-User line in your access log proves one thing: that user-agent requested that URL. It does not prove that OAI-SearchBot indexed the page, that the page ranked, or that an answer cited it.
Then let the change settle. OpenAI says its Search systems can take about 24 hours to adjust after a robots.txt update. A test ten minutes later measures your patience, not your policy.
The operator check
Write three answers before you open the file:
- May this public content be considered for foundation-model training?
- May it be surfaced inside ChatGPT search answers?
- Does the page need real access control when a person asks a tool to fetch it?
The first answer maps to GPTBot, the second to OAI-SearchBot, the third to your authentication design. Then test representative URLs against the complete robots.txt, and read the access log a day later.
None of this buys a citation. The citation-readiness audit tests whether one eligible page can be extracted and checked. How to get cited by ChatGPT covers the wider source work. If the door is open and the name still does not appear, Why am I not mentioned in ChatGPT? is the diagnosis. The Findability Engine owns the longer treatment; the book is in production and not for sale.
The crawler policy only decides which door stands open.