OAI-SearchBot: how OpenAI's search crawler affects your place in ChatGPT search

OAI-SearchBot is the one OpenAI crawler that decides whether ChatGPT search can use your pages at all, so an accidental block undoes the rest of your AI search optimization. This page covers who runs it, its exact user-agent strings, how it reads robots.txt, what allowing or blocking changes, how to confirm it in your logs and spot blocked requests, and OpenAI's own documentation.

Who runs OAI-SearchBot and what is it for?

OpenAI runs OAI-SearchBot to find and read pages for ChatGPT search. It's a search crawler (a bot that visits pages so a search product can show them), not a training crawler. OpenAI's crawler page says "OAI-SearchBot is for search" and recommends "allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges" (OpenAI).

OpenAI's other bots do different jobs, and confusing them is the most common mistake:

Source: OpenAI crawler documentation, as of October 2026. Our guide to AI crawlers compares OpenAI's bots with other operators' crawlers, such as Anthropic's ClaudeBot.

What is OAI-SearchBot's user-agent string and robots.txt token?

OAI-SearchBot's robots.txt token is OAI-SearchBot, and its user-agent string (the name it sends with each request) starts like a desktop Chrome browser. As of October 2026, OpenAI publishes this example, noting "the version number may change" (OpenAI):

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

When it fetches robots.txt, OpenAI says it "may use a user-agent string with an additional robots.txt marker", published as:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

Because the string looks like a browser at the start, firewall rules, bot filters and log queries should match the OAI-SearchBot token, not the browser part. The marker version tells you OpenAI re-read your robots.txt, which is useful after you change it.

Does OAI-SearchBot obey robots.txt?

Yes. OAI-SearchBot is one of the two OpenAI bots controlled through robots.txt (the text file at your site's root that tells crawlers which paths they may fetch). OpenAI's guidance for advertisers says "OpenAI crawlers respect these rules. If access is disallowed in robots.txt, crawling will stop immediately" (OpenAI Help Center).

Changes aren't reflected straight away. OpenAI says "for search results, please note it can take ~24 hours from a site's robots.txt update for our systems to adjust" (OpenAI). That matches the robots.txt standard, RFC 9309, which says crawlers "SHOULD NOT use the cached version for more than 24 hours". So after a fix, wait a day before judging it.

One more rule from RFC 9309: a crawler obeys the group that names it, and falls back to User-agent: * only "if no matching group exists". A site-wide Disallow: / under * therefore blocks OAI-SearchBot unless you give it its own group.

Should you allow or block OAI-SearchBot?

Allow OAI-SearchBot if you want your pages to be eligible for ChatGPT search; block it only for pages you want kept out. OpenAI's help article on web search says: "To make a website eligible for inclusion, allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses", and adds that "placement is not guaranteed" (OpenAI Help Center).

The firewall row is the one people miss. The same advertiser guidance lists three layers to check: robots.txt, "web protection and bot mitigation", and "human verification and anti-bot logic" such as CAPTCHAs and JavaScript challenges (OpenAI Help Center). It also flags rate limiting: look for "429 Too Many Requests" responses.

You can run a free check of which pages Google has indexed while you review your bot rules; ChatGPT search isn't the only place your pages need to be found.

Where ChatGPT search gets results besides OAI-SearchBot

OAI-SearchBot isn't ChatGPT search's only input. When OpenAI launched ChatGPT search on October 31, 2024, it said it "leverages third-party search providers, as well as content provided directly by our partners" (OpenAI), and its help center says ChatGPT search "sometimes partners with other search providers" (OpenAI Help Center). Microsoft announced in May 2023 that "Bing is coming to ChatGPT as the default search experience" (Microsoft). That's a 2023 statement, and OpenAI hasn't confirmed Bing's current role, so don't treat Bing, or any other index, as a known pipeline into ChatGPT; Bing Webmaster Tools shows what Bing itself has crawled. Our page on Bing, IndexNow and AI answers covers that side.

Which robots.txt rules allow OAI-SearchBot?

Give OAI-SearchBot its own group so a catch-all rule can't block it by accident. We ran each recipe through Python's standard robots.txt parser on October 17, 2026; the results are noted.

Allow ChatGPT search, opt out of training (OAI-SearchBot allowed, GPTBot blocked):

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Rescue a site-wide block. A staging-style Disallow: / under * blocks every crawler without its own group. Adding an OAI-SearchBot group lets it in while everything else stays blocked:

User-agent: *
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Keep private sections out of ChatGPT search (OAI-SearchBot can crawl /pricing, not /internal/ or /checkout/):

User-agent: OAI-SearchBot
Disallow: /internal/
Disallow: /checkout/

If you're on WordPress, check where your robots.txt actually comes from first; the WordPress robots.txt guide covers virtual and plugin-generated files. Then check your robots.txt for OAI-SearchBot and other AI crawlers to confirm which group each bot follows.

How do you verify OAI-SearchBot in your server logs?

Search your access logs for the OAI-SearchBot token, check which status codes it got, and confirm the IP addresses are OpenAI's. OpenAI publishes OAI-SearchBot's IP ranges at openai.com/searchbot.json. As of October 17, 2026, the file lists 39 IPv4 ranges and is dated January 2, 2026 (its creationTime field). The commands below assume the combined log format that nginx and Apache use by default; we ran them on a test log on October 17, 2026.

1. Count the status codes OAI-SearchBot received. Anything other than 200 or 304 is worth a look:

grep "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -c

2. List the URLs it couldn't fetch, with the code:

grep "OAI-SearchBot" access.log | awk '$9 != 200 && $9 != 304 {print $9, $7}' | sort | uniq -c | sort -rn

A cluster of 403s points to your firewall, CDN or bot protection; 429s point to rate limiting.

3. Check it re-read robots.txt after a change (the marker string):

grep "OAI-SearchBot" access.log | grep '"GET /robots.txt' | tail -5

4. Confirm the IPs are OpenAI's. This lists each address and tests it against the live JSON with Python's standard library:

grep "OAI-SearchBot" access.log | awk '{print $1}' | sort -u > searchbot-ips.txt
curl -s https://openai.com/searchbot.json -o searchbot.json
python3 - <<'EOF'
import ipaddress, json
nets = [ipaddress.ip_network(p["ipv4Prefix"]) for p in json.load(open("searchbot.json"))["prefixes"] if "ipv4Prefix" in p]
for line in open("searchbot-ips.txt"):
    ip = ipaddress.ip_address(line.strip())
    print(ip, "OpenAI" if any(ip in n for n in nets) else "NOT in searchbot.json")
EOF

An IP that isn't in the file is not OpenAI's search crawler, whatever its user agent says. OpenAI's advertiser guidance also warns against "relying solely on short-term IP observations from logs": allowlist from the published file and refresh it, rather than copying addresses you happened to see.

Where are OpenAI's official OAI-SearchBot docs?

OpenAI documents OAI-SearchBot on Overview of OpenAI Crawlers, with two help-center articles adding detail on eligibility and firewall setup.

For what happens after a page is crawled, see how AI search chooses its sources and the wider ChatGPT SEO playbook. To see whether ChatGPT actually mentions you, use AI visibility tracking; crawler logs can't show citations.

Allowing OAI-SearchBot makes your pages eligible for ChatGPT search; whether they're shown is OpenAI's call. FreshlyCited doesn't submit to ChatGPT: ChatGPT, Perplexity and other assistants use their own crawlers and search partners, and we don't claim to reach them. What it does is the Google side: it checks whether Google has indexed each URL, pushes only the ones it hasn't seen, and re-checks on day 1, 3 and 7, which makes good pages eligible for Google's AI Overviews and AI Mode. For sites you verify with an IndexNow key (an open "this page changed" notice that Bing and other participating engines accept; Google doesn't use it), it also sends the change to Bing, free. Check your pages free.