OAI-SearchBot allowed but still blocked? Check the path after robots.txt

Allowing OAI-SearchBot in robots.txt does not prove that it receives your article. robots.txt states what a crawler may fetch. What a request actually gets back is decided further along the path, by redirects, CDN settings, firewall rules and your application’s response. A page missing from ChatGPT search is a reason to investigate access, not proof that access is blocked.

OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT’s search features. OpenAI’s crawler overview recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. Test both, then inspect what the server sends back. A request carrying the crawler’s name needs an IP check; a 200 response needs a body check.

The 20-minute test at the end follows one URL from permission to delivery, using your CDN and origin logs.

If you are still deciding which OpenAI bot to allow at all, start with GPTBot vs OAI-SearchBot vs ChatGPT-User.

What OpenAI documents

OpenAI’s crawler overview and its publisher FAQ, both checked on 25 September 2026, say the following:

• Any public website can appear in ChatGPT search. Summaries and snippets require that you do not block OAI-SearchBot.
• Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.
• After a robots.txt change, OpenAI’s systems can take about 24 hours to adjust.
• The documented example user-agent string contains OAI-SearchBot/1.4. OpenAI notes that the version number may change and that robots.txt fetches may include an extra robots.txt marker.
• The published IP ranges are listed at openai.com/searchbot.json.

None of this promises a crawl date, a crawl frequency or a citation. The roughly 24-hour adjustment applies to robots.txt changes; it is not a timetable for clearing a firewall block.

The delivery path, layer by layer

Work down this table for one exact URL. Fix the first failure you find, then repeat the checks through to the response body. Here, a challenge is a check page served in place of the content, to test whether a person or a real browser sent the request.

Layer	What goes wrong	How to check
1. Host	robots.txt allows the crawler on www, but the article lives on the bare domain or a docs. subdomain with its own file	Open /robots.txt on every host the URL touches
2. Redirects	The chain ends at the homepage, a login page, a consent page, or a host that disallows the crawler	Follow the chain with curl and write down every hop
3. robots.txt response	The file itself returns a server error, or a challenge instead of the rules	Check both its status code and its body
4. CDN bot settings	An “AI bots” or bot-management setting blocks or challenges the crawler	Read the security events for that URL and user agent
5. Firewall rules	A country block, a rate limit (status 429), or a custom rule fires before any allow setting	Match the logged request to a rule ID
6. The response	Status 200, but the body is a challenge, an error template, a login form, or an empty shell	Save the body and search it for a sentence from the article

For layer 1, RFC 9309, the robots.txt standard, puts the file at the top level of each host. Check www.example.com and example.com separately: an allow rule on one says nothing about the other.

For layer 3, the same standard says that when robots.txt is unreachable because of server or network errors, including a 5xx status, the crawler “MUST assume complete disallow”. A 4xx status such as 404 counts as unavailable, and the standard then allows a crawler to fetch anything on the host. OpenAI does not document how it handles either case; the RFC explains the difference but does not predict what OpenAI does. If robots.txt itself returns a challenge page, check your articles for the same problem.

For layer 4, look at the CDN’s bot settings. Cloudflare, for example, lets site owners block AI bots by behavior: Search, Agent or Training. Its documentation announced new defaults for new domains from 15 September 2026: Training and Agent bots blocked on pages that show ads, Search left allowed. Your own domain may be set differently, so check it. Cloudflare’s verified bots page names OAI-SearchBot as its example for the legacy AI Search category, so also check which categories your rules act on.

For layer 5, check the rule order. Cloudflare’s troubleshooting page warns that a crawler set to Allow can still be blocked by an earlier custom firewall rule, and that such blocks “may not be visible in AI Crawl Control analytics”.

Check country blocks too. Vercel and MERJ observed the crawlers they attributed to ChatGPT running from US data centers in late 2024. That is a third-party observation, not OpenAI documentation of where its crawler runs today. A Europe-only rule is worth checking, but only your logs can show whether it blocked the crawler.

Google states the same principle for its own AI features: crawling must be allowed “in robots.txt, and by any CDN or hosting infrastructure”, according to its AI features page.

Three commands that show what the path returns

Replace the example URL with your page and the placeholder with a sentence from the article.

# 1. Every hop, with its status code and Location header
curl -sSL -D - -o /dev/null 'https://example.com/your-page/' | grep -iE '^(HTTP|location)'

# 2. The robots.txt status on each host you saw in step 1
curl -sS -o /dev/null -w '%{http_code}\n' 'https://example.com/robots.txt'

# 3. The final page, requested with the crawler's name
curl -sSL -o page.html -w '%{http_code} %{num_redirects} %{url_effective}\n' \
  -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' \
  'https://example.com/your-page/'
grep -c -F 'a sentence from the article' page.html

The curl manual documents these options. The second command reports only a status, so open robots.txt separately to read its rules. The third saves the final body as page.html. Open it even if the sentence matches: the match may sit in a meta tag, not in the article. If the count is zero, check whether HTML markup or a line break splits the sentence before you conclude that the text is missing.

The third request uses the crawler’s name but still comes from your machine, so treat it as a local probe. It can expose a rule keyed on the user agent; it cannot show what OpenAI’s network receives. A firewall that verifies bots may reject it precisely because the claimed name does not match the source IP.

Is that request really OAI-SearchBot?

Anyone can send the OAI-SearchBot user-agent string. For a logged request carrying that name, check the client IP against OpenAI’s published ranges. The script tests range membership only, not what the crawler received:

import ipaddress, json, sys, urllib.request

ranges = json.load(urllib.request.urlopen("https://openai.com/searchbot.json"))
ip = ipaddress.ip_address(sys.argv[1])
hit = any(ip in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
          for p in ranges["prefixes"])
print(ip, "is in OpenAI's published OAI-SearchBot ranges" if hit else "is NOT in the published ranges")

Save it as check_ip.py and run python3 check_ip.py 203.0.113.7, replacing the example address with the client IP from your log. On 25 September 2026 the file listed 39 IPv4 ranges. The script downloads a fresh copy on every run; an allowlist in your firewall needs the same refresh.

Two traps catch rules that are meant to let the crawler in:

• Matching one exact string, version number included. OpenAI says the version may change, and an exact-match rule can then stop matching.
• Allowing by name alone. Any scraper can put “OAI-SearchBot” in a request. Require the name and the published IP range together.

Perplexity’s crawler page describes that pattern for its own bots: combine the user agent with the published IP ranges, allow matching requests ahead of blocking rules, and refresh the IP list automatically. That is guidance for Perplexity’s crawlers, not OpenAI’s; for OAI-SearchBot, use OpenAI’s list.

Read the logs without asking too much of them

Your CDN log and your origin log (the log on your own server) cover different parts of the path. A request blocked at the CDN never reaches the origin, so an empty origin log cannot rule out an attempt. Search both for OAI-SearchBot; that also finds robots.txt fetches with the extra marker.

For each request you inspect, record the time, host, path, status, IP-check result and any blocking rule ID. Note how long the logs are kept and whether they are sampled. Missing entries are a gap in the evidence, not proof that the crawler stayed away. A log line with only a status also leaves the response body unverified.

A worked example (hypothetical)

Imagine a training company in Amsterdam. Its robots.txt allows OAI-SearchBot on www. The team has not seen its course pages in ChatGPT search, and the origin log contains no OAI-SearchBot requests.

The team checks the hosts and redirects, then confirms that robots.txt returns 200 with the intended rules. The CDN’s security events show requests carrying the OAI-SearchBot name, from addresses that pass the IP check, answered with 403 by a rule called “EU and UK only”.

The team adds an exception, placed before the country block, that requires both the OAI-SearchBot user agent and an IP in OpenAI’s published ranges, with the list refreshed automatically. The country block stays in place for all other traffic. On a later request, the CDN logs a 200 and the request appears in the origin log. That establishes that this request passed the block. Before closing the check, the team still needs evidence that the response contained the course text.

Whether ChatGPT then names the company is a separate test. Why am I not mentioned in ChatGPT? covers that diagnosis.

What a green check does not prove

A response that contains the article establishes delivery for that request. It does not establish selection, quotation or citation; How to get cited by ChatGPT covers that work. If the article text appears only after JavaScript runs in a browser, the crawler may still receive an empty shell. That is a rendering problem rather than an access problem, and only the returned document shows it.

Try this today (20 minutes)

1. Minutes 0–2: Pick one public URL you want available to ChatGPT search, and one distinctive sentence from its article.
2. Minutes 2–7: Run the curl commands. Record every host and status, then open page.html and look for the sentence. Label the result a local probe.
3. Minutes 7–11: Open robots.txt on every host in the chain. Check its status, its body and its rules for OAI-SearchBot at the relevant path.
4. Minutes 11–18: Search the last seven days of CDN and origin logs for OAI-SearchBot. Check the IPs of up to three relevant hits against the published ranges, and record their statuses and any rule IDs. If there are no hits, record that and the period the logs cover.
5. Minutes 18–20: Write down the finding for that URL. If a verified request was blocked, name the layer and the rule. If it got a 200, note whether you have evidence of the body. If all you have is a successful local probe, leave crawler delivery unconfirmed.

The result is a concrete record: the URL, the request time, the checked IP, the status, any blocking rule and what you know about the returned body. That gives the next person either a failure to fix or a precise gap to investigate.

Cite this:OAI-SearchBot allowed but still blocked? Check the path after robots.txt.Len P. van der Hof. https://lenvanderhof.com/en/blog/oai-searchbot-access-check/ · Published 21 October 2026.