GPTBot: what OpenAI's training crawler does and how to control it

That split matters for AI search optimization, because the training decision and the search decision are made by different bots, and many sites block one when they meant the other. This page covers who runs GPTBot, its exact user-agent string, how it reads robots.txt, what allowing or blocking changes, how to confirm real GPTBot traffic in your logs, and where OpenAI documents it.

Who runs GPTBot and what is it for?

GPTBot is run by OpenAI to gather web content for training its foundation models. It is a training crawler (a bot that visits pages to read and copy them), not a search crawler and not a fetcher acting for a user. OpenAI's crawler page says GPTBot "is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models" (OpenAI).

Three OpenAI bots matter to most site owners, and each has a different job:

Source: OpenAI crawler documentation, as of October 2026. For how other AI companies split the same jobs, see the AI crawler directory and Anthropic's ClaudeBot.

What is GPTBot's user-agent string and robots.txt token?

GPTBot identifies itself with the token GPTBot inside its user-agent string (the name a bot sends with every request). As of October 2026, OpenAI publishes this example and notes that "the version number may change" (OpenAI):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Three details help when you read logs or write rules:

1. The robots.txt token is GPTBot. Match on that, not on the full string, because the version number changes.
2. There's a robots.txt marker. OpenAI says that "when fetching robots.txt files, we may add a robots.txt marker to the user-agent string" so you can tell those requests apart when your logs don't record paths.
3. The info URL is https://openai.com/gptbot. It's part of the string, which makes GPTBot easy to search for, and also easy for anyone to copy (see the log section below).

Does GPTBot obey robots.txt?

Yes. OpenAI lists GPTBot as one of the two bots it manages through robots.txt (the plain text file at your site's root that tells crawlers which paths they may fetch). OpenAI says it "uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI" (OpenAI).

Two rules from the robots.txt standard, RFC 9309, trip people up with GPTBot:

• A named group replaces the catch-all. A crawler obeys the group that names it. RFC 9309 says crawlers fall back to the User-agent: * group only "if no matching group exists". Once you add User-agent: GPTBot, GPTBot stops reading your User-agent: * rules.
• robots.txt is a request, not a lock. The standard says its rules "are not a form of access authorization." A well-behaved crawler follows them; anything that only pretends to be GPTBot won't. Enforcement needs a server or firewall rule.

We checked the first rule by running the recipes below through Python's standard robots.txt parser on October 17, 2026: with a GPTBot group present, GPTBot was allowed into a folder that only the * group disallowed.

Should you allow or block GPTBot?

Block GPTBot if you don't want your content used for OpenAI model training; allow it if you're comfortable with that use. It's a licensing and content decision, not a search-visibility decision. OpenAI says "disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models" (OpenAI).

The independence is explicit. OpenAI gives the example that "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" (OpenAI). It also says that if both are allowed, "we may use the results from just one crawl for both use cases to avoid duplicative crawling", so you may see fewer GPTBot visits than you expect.

What blocking GPTBot can't do: remove what OpenAI has already collected, or change what other AI companies crawl. It also doesn't touch the retrieval side of LLM SEO, where answers draw on pages fetched at question time rather than on training data. Each operator has its own token; how AI search chooses its sources compares them.

If your real goal is being found in search and AI answers, the step after your bot rules is checking that your pages are indexed (stored in a search engine's index, the library answers draw on). You can check which of your pages Google has indexed for free.

Which robots.txt rules block GPTBot without touching ChatGPT search?

Name GPTBot in its own group and leave OAI-SearchBot allowed. Each recipe below was run through a robots.txt parser on October 17, 2026, with the result noted.

Block training, keep ChatGPT search (GPTBot blocked everywhere; OAI-SearchBot allowed):

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Keep a few sections out of training (GPTBot can crawl the blog, not /members/ or /drafts/):

User-agent: GPTBot
Disallow: /members/
Disallow: /drafts/

Don't lose your catch-all rules. This version lets GPTBot into /admin/, because the GPTBot group replaces the * group:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /members/

Repeat the catch-all paths inside the GPTBot group to fix it:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /admin/
Disallow: /members/

On WordPress, where robots.txt may be virtual or plugin-generated, see editing robots.txt on WordPress before you paste these in. Afterwards, check which AI crawlers your robots.txt allows, since a new GPTBot group replaces your * rules for that bot.

How do you verify GPTBot in your server logs?

Find requests whose user agent contains GPTBot, then confirm each IP address is on OpenAI's published list. The user-agent string alone isn't enough, because anyone can send it. OpenAI publishes GPTBot's IP ranges at openai.com/gptbot.json. As of October 17, 2026, that file lists 18 IPv4 ranges and is dated September 22, 2026 (its creationTime field), so fetch it fresh rather than hard-coding addresses.

1. List the IPs claiming to be GPTBot, with a count for each (combined log format, as nginx and Apache write by default):

grep -i "GPTBot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn

2. See how often it visits, per day:

grep -i "GPTBot" access.log | awk '{print substr($4, 2, 11)}' | sort | uniq -c

3. Check whether it read your robots.txt after you changed it:

grep -i "GPTBot" access.log | grep -c '"GET /robots.txt'

4. Check each IP against OpenAI's list. Save this as check_openai_ip.py; it reads the live JSON and tests one address:

import ipaddress, json, sys, urllib.request

bot, ip = sys.argv[1], ipaddress.ip_address(sys.argv[2])
url = f"https://openai.com/{bot}.json"
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (ip-range-check)"})
data = json.load(urllib.request.urlopen(req))
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in data["prefixes"]]
found = any(ip in net for net in nets)
print(f"{ip} {'is' if found else 'is NOT'} in {url} (list dated {data['creationTime'][:10]})")

5. Run it for every GPTBot IP in the log:

for ip in $(grep -i "GPTBot" access.log | awk '{print $1}' | sort -u); do
  python3 check_openai_ip.py gptbot "$ip"
done

When we ran this on October 17, 2026 against a test log, an address from OpenAI's range returned "is in https://openai.com/gptbot.json (list dated 2026-09-22)" and a spoofed request from an unrelated address returned "is NOT in". Treat the "is NOT" lines as impersonators: they don't follow OpenAI's rules, so block them at the server or firewall, not in robots.txt.

Where are OpenAI's official GPTBot docs?

OpenAI documents GPTBot on its crawler overview page, Overview of OpenAI Crawlers, which also covers OAI-SearchBot, OAI-AdsBot and ChatGPT-User. The older platform.openai.com/docs/bots address redirects there as of October 2026.

The page ends with "Subscribe to the RSS feed for updates to this page", the simplest way to hear about a new version number or a new bot. For the search side of the same decision, see ChatGPT SEO.

Blocking or allowing GPTBot is a one-line choice about training. Being found in search and AI answers is a separate job that starts with your pages being indexed. FreshlyCited's index checker shows whether Google has indexed each URL, and smart push sends only the ones it hasn't seen, then re-checks them on day 1, 3 and 7, so good pages become eligible for Google's AI Overviews and AI Mode. Google still decides what it indexes. We don't submit to ChatGPT, and we don't claim to. Check your pages free: 100 checks a month, no card.