AI crawlers: who runs each bot, what it collects and how to control it

Block a search crawler by mistake and that company's AI answers can't use your pages, however good they are. This page lists the main AI and search crawlers as of October 2026, with each operator's own words on robots.txt, then covers which ones matter for AI answers, three tested robots.txt recipes, how to verify a bot in your logs, and what robots.txt can't do.

What are AI crawlers, and what are the three kinds?

An AI crawler (also called an LLM crawler or AI bot) is an automated client run by an AI company that requests pages from your site to collect training data, to build a search index, or to fetch a page for a person in real time. Like any crawler (a bot that visits pages to read them), it names itself in a user-agent string, and most operators publish a short token you can address in robots.txt.

Some agents mix jobs. Apple says Applebot's data "may also be used to help train Apple foundation models", and the training opt-out is the separate Applebot-Extended token (Apple). Meta-ExternalAgent crawls "for use cases such as training foundation AI models or improving products by indexing content directly" (Meta). Amazonbot "may be used to train Amazon AI models" as well as improve Amazon's products (Amazon).

Two entries aren't crawlers at all. Google says "Google-Extended doesn't have a separate HTTP request user agent string" (Google), and Apple says "Applebot-Extended does not crawl webpages" (Apple). You'll never see either in your logs. They are robots.txt tokens that tell Google and Apple how content their main crawlers collected may be used.

What's on the AI crawler list as of October 2026?

The table lists each crawler's robots.txt token, operator, purpose and robots.txt behavior, checked on the operator's own documentation on October 17, 2026. Names link to our page on that bot where one exists; quotes are the operator's words.

Not listed: Bytespider (ByteDance). We found no documentation for it on ByteDance's own sites, so we can't state its purpose or robots.txt behavior from a primary source.

Three details from these pages catch people out:

• Applebot borrows Googlebot's rules. Apple says "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions" (Apple).
• Crawl-delay support varies. Anthropic and Common Crawl support it; Google lists it as unsupported (Google), and Apple and Amazon say they don't follow it.
• Changes aren't immediate. OpenAI says it "can take ~24 hours" for search systems to reflect a robots.txt update, and Perplexity, Meta and Amazon each give up to about 24 hours.

Which AI crawlers decide whether you can appear in AI answers?

Search crawlers decide whether your pages can appear in AI answers: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, and Googlebot and Bingbot for Google's and Microsoft's AI features. Training crawlers only collect data for future models, and blocking them doesn't remove you from search-based answers.

The operators say so directly:

• OpenAI: a webmaster "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" (OpenAI). Our GPTBot and OAI-SearchBot pages cover each side, and how ChatGPT search picks its sources covers the rest.
• Google: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google).
• Apple: "Webpages that disallow Applebot-Extended can still be included in search results" (Apple).
• Anthropic: disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results", while restricting ClaudeBot signals that future content "should be excluded from our AI model training datasets" (Anthropic). Our guide to getting cited by Claude goes further.

Google and Microsoft have no separate AI search crawler. For AI Overviews and AI Mode, Google says a page "must be indexed and eligible to be shown in Google Search with a snippet" and that "robots.txt directives for Googlebot is the control" for crawl access (Google). Indexed means in the search engine's index, the library it answers from. Microsoft says Copilot and Copilot Chat "use generated search queries sent to the Bing search service to ground responses in web data" (Microsoft Learn). So the classic search crawlers are AI crawlers too, and an old Disallow aimed at Googlebot or Bingbot reaches straight into AI answers.

The quieter mistake is a broad User-agent: * block, CDN bot rule or firewall that stops OAI-SearchBot or PerplexityBot without anyone deciding to. For how each engine turns crawled pages into answers, read how AI search chooses its sources.

If you've just unblocked a search crawler and want to know which of your key pages Google has indexed, run a free index check on them.

How do I block AI training crawlers but stay in AI search?

To opt out of AI training while staying in AI search, give the training tokens their own Disallow: / group and leave search crawlers and user fetchers under your normal rules. Each recipe below uses syntax from RFC 9309 and Google's robots.txt documentation, and we ran all three through Protego, an open-source Python robots.txt parser, on October 17, 2026.

Two rules make these work. Several User-agent lines can share one group. And a crawler obeys only one group: Google says "Only one group is valid for a particular crawler" and "User agent specific groups and global groups (``) are not combined"* (Google robots.txt spec). RFC 9309 adds that matching is case-insensitive and that the "most specific match found MUST be used" for paths (RFC 9309).

Recipe 1: opt out of training, stay open to search

# Opt out of AI training; stay open to search crawlers and user fetchers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Tested result: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot are disallowed everywhere. Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and the user fetchers are allowed. Amazonbot is left open here because Amazon describes it first as improving its products and services; add it to the first group if you also want to opt out of its possible training use. Blocking Meta-ExternalAgent may also limit Meta's indexing, since Meta lists both uses for it.

Recipe 2: keep a members area out of AI search too

Once you give a bot its own group, it stops reading the * group, so any rule you want it to follow must be repeated there.

# Opt out of AI training; keep AI search crawlers out of /members/;
# keep /account/ closed to every crawler
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /members/
Disallow: /account/

User-agent: *
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

Tested result: OAI-SearchBot, Claude-SearchBot and PerplexityBot can crawl the site but not /members/ or /account/; Googlebot and Bingbot can crawl /members/ but not /account/. When we dropped Disallow: /account/ from the named group, the parser let all three AI search crawlers into /account/. User fetchers fall under * here, and some of them may ignore it anyway (see below), so real member content still needs a login.

Recipe 3: stay in Google and Bing, opt out of AI-specific crawlers

# Search engines only: keep Google and Bing, opt out of AI-specific crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Tested result: Googlebot, Bingbot and Applebot are allowed; the listed AI crawlers and tokens are not. The trade-off: this takes you out of ChatGPT search, Claude's search and Perplexity, but not out of AI Overviews, AI Mode or Copilot, which run on Googlebot and Bingbot. User fetchers aren't covered either.

On WordPress, where robots.txt is usually generated rather than a file you edit, see editing robots.txt in WordPress for where these rules go. To test a live file against every crawler token at once, use the script in how to check your robots.txt for AI crawlers.

How do I verify an AI crawler in my server logs?

To verify a crawler, match the request's IP address against the operator's published IP list or run a reverse and forward DNS check; never trust the user-agent string alone. Bing puts it plainly: "user agent strings are easy to spoof" (Bing).

Reverse DNS for Googlebot and Bingbot

Google's steps are to run "a reverse DNS lookup on the accessing IP address from your logs", check the domain, then run a forward lookup and confirm it returns the same IP (Google). Bing's are the same with search.msn.com (Bing). We ran both on October 17, 2026:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

host 157.55.39.1
# 1.39.55.157.in-addr.arpa domain name pointer msnbot-157-55-39-1.search.msn.com.
host msnbot-157-55-39-1.search.msn.com
# msnbot-157-55-39-1.search.msn.com has address 157.55.39.1

Published IP lists for OpenAI, Anthropic, Perplexity and others

The JSON files above share one shape (a prefixes list of ipv4Prefix or ipv6Prefix ranges), so one script checks any of them. It returned "match" for addresses inside the GPTBot, Claude, PerplexityBot, Bingbot and CCBot lists, and "no match" for 8.8.8.8, when we ran it on October 17, 2026:

# usage: python3 check_ip.py <ip> <json-url>
import ipaddress, json, sys, urllib.request

ip, url = ipaddress.ip_address(sys.argv[1]), sys.argv[2]
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (ip-check)"})
with urllib.request.urlopen(req) as r:
    prefixes = json.load(r)["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
print("match" if any(ip in n for n in nets) else "no match")

Fetch the lists fresh rather than hard-coding them. Bing asks you to "Refresh this list daily as we can change this list any time" (Bing). Bing Webmaster Tools also shows Bingbot's view of your site; see how to use Bing Webmaster Tools.

What can't robots.txt do?

robots.txt asks well-behaved crawlers to stay out; it can't enforce anything, it doesn't remove pages from an index, and it doesn't reliably cover fetches a person starts. Google says the instructions "cannot enforce crawler behavior to your site; it's up to the crawler to obey them" (Google).

• It isn't access control. RFC 9309 says the rules "are not a form of access authorization." Use a login for anything private.
• It isn't a noindex. Google calls robots.txt "not a mechanism for keeping a web page out of Google" and points to noindex or password protection instead (Google).
• User-initiated fetchers may not follow it. OpenAI (ChatGPT-User), Perplexity (Perplexity-User), Meta (Meta-ExternalFetcher), Google (user-triggered fetchers) and Amazon (Amzn-User) all say so in the table above. Anthropic is the exception: it says its bots honor robots.txt, and that Claude-User "allows site owners to control which sites can be accessed".
• Blocking IPs can backfire. Anthropic warns that blocking its IP addresses "may not work correctly" as an opt-out, because it stops Anthropic reading your robots.txt (Anthropic).
• Server errors count as a block. Under RFC 9309, if robots.txt is unreachable because of a server error, "the crawler MUST assume complete disallow."

llms.txt is not a crawler control either. It's a proposed Markdown summary of your site for AI tools, and none of the crawler documentation pages linked above names it as a way to allow or block a bot; they document robots.txt tokens. Our llms.txt explainer covers what it does do.

What nobody can promise, and what you can control

Nobody can promise that allowing a crawler gets your page cited. Allowing OAI-SearchBot makes you eligible for ChatGPT search, and being in Google's index makes a page eligible for AI Overviews, but each engine still chooses its sources on quality, relevance and authority.

What you control:

1. Access: robots.txt, CDN and firewall rules that let the search crawlers you want reach your pages.
2. Training choices: separate tokens, set deliberately rather than by a blanket block.
3. Indexing: whether your important pages are in Google's and Bing's indexes, in their current version. See how to get Google to crawl your site; for Bing, IndexNow (an open "this page changed" notice that Bing and other participating engines accept; Google doesn't use it) is the direct route.
4. Content: pages worth citing, kept current, which the AI search optimization hub covers.

Where FreshlyCited fits

FreshlyCited covers the indexing step for Google, with Bing as a free extra, not the AI crawlers in the table. Google's AI Overviews and AI Mode only link to indexed pages, so that's where it starts. It checks whether Google has indexed each URL ("Seen by Google" or "Invisible"), pushes only the ones Google hasn't seen, and re-checks on day 1, 3 and 7. For sites you verify with an IndexNow key, it also sends the change through IndexNow ("Sent to Bing"), free. ChatGPT, Claude, Perplexity and other assistants use their own crawlers and search partners. We don't submit to them, and we don't claim to. Google decides what it indexes.

Fix your robots.txt first, then find out which of your important pages Google actually has. Check your pages free: the free tier gives you 10 free links to try, then 100 free checks a month, no card needed.