ClaudeBot: Anthropic's training crawler, explained

That split matters because the three bots feed different things: training data, Claude's search results, and pages a person asks Claude to read. This page covers who runs ClaudeBot, its user-agent string, how it treats robots.txt, what allowing or blocking it changes, how to confirm a visit in your logs, and where Anthropic documents it.

Who runs ClaudeBot, and what is it for?

ClaudeBot is run by Anthropic, the company behind Claude, and its job is collecting training data. Anthropic's crawler help article says ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training" (Anthropic Help Center).

The same article, dated April 7, 2026 as of October 2026, lists two other bots with different jobs:

So ClaudeBot is the training crawler, Claude-SearchBot is the search crawler, and Claude-User is the on-demand fetcher. The same pattern exists at OpenAI, where GPTBot is the training crawler, OAI-SearchBot handles ChatGPT search and ChatGPT-User fetches pages for users. For how all of these bots fit together, see the AI crawlers guide, and for the wider picture of being found in AI answers, the AI search optimization guide.

Anthropic's model documentation describes the crawler's limits too. Its Claude Opus 4.5 system card (November 2025) says "We do not access password-protected pages or those that require sign-in or CAPTCHA verification" (Anthropic system card).

What is ClaudeBot's user-agent string and robots.txt token?

ClaudeBot's robots.txt token is ClaudeBot, the name Anthropic uses in every robots.txt example in its help article. The robots.txt standard, RFC 9309, says crawlers must match that name case-insensitively, but copying Anthropic's spelling exactly is the safe choice.

Anthropic's help article doesn't print the full user-agent header. Our own server log for freshlycited.com recorded ClaudeBot requests on October 16–17, 2026 with this exact header:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

The useful parts are ClaudeBot/1.0, the bot name and version, and claudebot@anthropic.com, the contact address. Match on ClaudeBot rather than the whole string, because version numbers and browser prefixes can change.

The other two bots use their own tokens: Claude-SearchBot and Claude-User. A rule for one doesn't apply to the others. Anthropic's crawler article mentions robots.txt only, not llms.txt; see what llms.txt does and doesn't do before adding one for AI bots.

Does ClaudeBot obey robots.txt and Crawl-delay?

Yes. Anthropic says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt" and "respect anti-circumvention technologies (e.g., we will not attempt to bypass CAPTCHAs for the sites we crawl.)" (Anthropic Help Center).

Anthropic also supports Crawl-delay, a rule the robots.txt standard doesn't define. In its own words: "To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt." Its example:

User-agent: ClaudeBot
Crawl-delay: 1

Use Crawl-delay if ClaudeBot's request rate strains your server but you still want to allow it. Anthropic doesn't state the unit or how strictly it applies the value, so treat it as a request to slow down.

Two rules from the robots.txt standard decide which group of lines applies to a crawler such as ClaudeBot:

1. If a group names the crawler, it follows that group, and the User-agent: * group doesn't apply to it.
2. If no group names it, "crawlers MUST obey the group with a user-agent line with the '' value, if present"* (RFC 9309).

The first rule catches people out. A User-agent: ClaudeBot group with only Crawl-delay in it doesn't inherit the Disallow lines from your * group, so repeat any paths you still want kept private.

Should you allow or block ClaudeBot?

Block ClaudeBot if you don't want your future content used for Anthropic's model training; allow it if you're comfortable with that. The decision doesn't touch Claude's search results or user fetches, which Claude-SearchBot and Claude-User handle under their own tokens.

The most common choice, out of training but still findable, looks like this. We tested it with Python's built-in robots.txt parser: ClaudeBot is refused everywhere, and the two Claude search bots are allowed except under /admin/.

# Opt out of Anthropic model training
User-agent: ClaudeBot
Disallow: /

# Keep Claude's search index and user fetches,
# but repeat private paths: this group replaces the * group
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /admin/

User-agent: *
Disallow: /admin/

Keep these points in mind:

• Blocking is forward-looking. Anthropic's wording is about "future materials". It says nothing about content already collected.
• Do it per subdomain. Anthropic says to add the rule "for every subdomain that you wish to opt out from", because each subdomain serves its own robots.txt.
• Don't rely on IP blocks. Anthropic says IP blocking "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."
• Blocking Claude-SearchBot has a visibility cost. If you want to be found in Claude's answers, that's the bot to keep allowed. The Claude SEO guide covers how Claude finds and cites pages.

If your site runs on WordPress, see editing robots.txt on WordPress for where the file lives and how plugins change it. To confirm the result, test your robots.txt for Anthropic's three bots and the other AI crawlers in one run.

Letting Claude's search bots in is only half the job. Search engines, and the AI answers built on them, can only surface pages they have crawled and indexed (added to the engine's index, the library it answers from). Check which of your pages Google has indexed before you tune rules for any one AI bot.

How do you verify ClaudeBot in your server logs?

Verify ClaudeBot by checking that the request's IP address is on Anthropic's published list. Anthropic's article links a JSON file of IP ranges and says "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic" (Anthropic Help Center). The user-agent string on its own isn't enough, because anyone can send it.

1. Find the requests. On a server with a standard combined access log (nginx or Apache), count ClaudeBot requests by IP:

   grep -i "claudebot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn

2. Download Anthropic's IP list. It's at claude.com/crawling/bots.json. On October 17, 2026 it held 38 IPv4 ranges and no IPv6 ranges, with a creationTime of October 7, 2026. It changes, so fetch it fresh rather than hard-coding it.
3. Check each IP against the list. This Python script, which we ran on October 17, 2026, fetches the list and tests the IPs you pass it:

   import ipaddress, json, sys, urllib.request

   req = urllib.request.Request("https://claude.com/crawling/bots.json", headers={"User-Agent": "ip-check"})
   data = json.load(urllib.request.urlopen(req))
   nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in data["prefixes"]]
   for arg in sys.argv[1:]:
       ip = ipaddress.ip_address(arg)
       found = any(ip.version == n.version and ip in n for n in nets)
       print(ip, "is on Anthropic's list" if found else "is NOT on Anthropic's list")

Run it as python3 check_anthropic_ip.py 216.73.216.218.

4. Don't count on reverse DNS. Google suggests a reverse DNS lookup as one way to confirm Googlebot (Google Search Central). Anthropic's article doesn't offer that method, and for the ClaudeBot IP in our own log a reverse lookup returned no hostname (NXDOMAIN), so use the IP list instead.
5. Treat mismatches as impostors. A request that says ClaudeBot but comes from an IP outside the list isn't from Anthropic, and you can block it like any other unwanted bot.

Here is what that looks like on our own site. Between 22:48 UTC on October 16, 2026 and 12:36 UTC on October 17, 2026, the access log for freshlycited.com recorded 62 ClaudeBot requests. All came from 216.73.216.218, inside the 216.73.216.0/22 range on Anthropic's list. Ten were fetches of robots.txt, nine were fetches of sitemap.xml, and none touched a path our robots.txt disallows. That's one site over about 14 hours, so it shows the pattern, not typical volumes. A redacted line from that log:

2026-10-16T22:48:46Z 216.73.216.218 GET /robots.txt 200 "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"

Visits from ClaudeBot tell you Anthropic collected the page for possible training. They don't mean Claude will cite it, and they say nothing about Google: for that, use the steps in getting Google to crawl your site. For what decides citations, see how AI search chooses its sources.

Where are Anthropic's official ClaudeBot docs?

Anthropic documents ClaudeBot in one help article and one IP file. Check them before changing your rules, since both are updated over time:

• Crawler help article: Does Anthropic crawl data from the web, and how can site owners block the crawler? covers the three bots, robots.txt, Crawl-delay and opting out. The article has a form to subscribe to updates.
• IP ranges: claude.com/crawling/bots.json.
• Training data practices: the data section of Anthropic's system cards, such as the Claude Opus 4.5 system card.
• Contact: Anthropic asks site owners with questions, or who think a bot is misbehaving, to email claudebot@anthropic.com from an address on the domain concerned, "as it is otherwise difficult to verify reports."

Make sure the pages you want found are indexed

Your robots.txt decides which Anthropic bots can read your site. Whether any search or AI engine can surface a page still depends on it being crawled and indexed. FreshlyCited checks whether Google has indexed each URL, asks Google to look at the ones it hasn't, and re-checks on day 1, 3 and 7; see how an index check works. Indexed pages are eligible for Google's AI Overviews and AI Mode. We don't submit to Claude or other assistants, and we don't claim to. You can start free with 100 checks a month and no card. Check my pages free.