How to check your robots.txt for AI crawlers

A crawler that robots.txt turns away can't read the page, so a rule written for one bot can quietly decide what a search engine or AI answer is able to use. This guide covers which crawlers to test, how they pick their rules, what Google's and Bing's tools show, a checker you can run, how to read the results and the mistakes that block the wrong bot.

Which crawlers should a robots.txt check cover?

A robots.txt check should cover the search crawlers first, then each AI company's search, user and training agents, because each one reads its own token. As of October 2026, these are the tokens the operators document, in the order Google comes first:

Sources: Google crawlers, OpenAI crawlers, Anthropic, Perplexity, Apple, Common Crawl. Our AI crawlers directory explains each one and gives ready-made rule sets; the pages on OAI-SearchBot, GPTBot and ClaudeBot go deeper.

How does a crawler decide which robots.txt rules apply?

A crawler picks one group of rules, then the longest rule in that group that matches the URL path. Google's robots.txt specification spells out each step, as of October 2026 (Google Search Central):

1. One group per crawler. "Only one group is valid for a particular crawler," chosen as "the group with the most specific user agent that matches." A crawler with its own group ignores the * group completely.
2. Its own groups merge. If two groups name the same crawler, their rules "are combined internally into a single group," but "User agent specific groups and global groups () are not combined."*
3. Longest path wins. Crawlers "use the most specific rule based on the length of the rule path," and on a tie "Google uses the least restrictive rule."
4. Only four fields count. Google supports user-agent, allow, disallow and sitemap; "other fields such as crawl-delay aren't supported."
5. Size and caching. Google reads the first 500 KiB, and "generally caches the contents of robots.txt file for up to 24 hours."
6. Errors. Google treats a 4xx response, except 429, as if no robots.txt existed; a 5xx makes it stop crawling the site for a while.

Point 1 is where most accidents happen. Add a User-agent: GPTBot group with one Disallow: /drafts/ line, and GPTBot no longer follows your Disallow: /private/ rule under *. In our test of exactly that file, GPTBot was allowed into /private/x.

Other crawlers follow the same standard (RFC 9309), but they adjust on their own schedule: OpenAI says search changes take "~24 hours", and Perplexity says changes "may take up to 24 hours."

What do Google's and Bing's robots.txt tools show?

Google's and Bing's tools show whether their own crawlers can read your file, not what AI crawlers will do.

• Google Search Console: the robots.txt report sits under Settings. It "shows which robots.txt files Google found for the top 20 hosts on your site, the last time they were crawled, and any warnings or errors encountered," and lets you "request a recrawl of a robots.txt file for emergency situations" (Search Console Help). Use it after any edit, so Google isn't working from a stale copy.
• Bing Webmaster Tools: the robots.txt tester checks a URL against the file and shows "which statement is blocking it and for which user agent," with a toggle "between Bingbot and AdIdxbot" (Bing Webmaster Tools help). Its editor lets you try changes before you upload them.

Neither tool takes GPTBot, ClaudeBot or PerplexityBot as an input. For the Google side of a single page, the URL Inspection tool also reports whether crawling was allowed; a page Google can't fetch shows in the Page indexing report as blocked by robots.txt.

If you've just changed robots.txt and want to know which of your pages Google has indexed since, check them free.

How do you test every AI crawler at once?

You test every AI crawler at once by fetching the live robots.txt and applying Google's rules for each token. This script does that with Python's standard library; we wrote and ran it on October 17, 2026:

# ai_robots_check.py: which search and AI crawlers may fetch a URL, by Google's robots.txt rules
# (one group per crawler: its own named group, else "*"; longest matching rule wins; Allow wins ties).
# Usage: python3 ai_robots_check.py https://example.com/some/page
import re, sys, urllib.request, urllib.parse

BOTS = ["Googlebot", "Google-Extended", "Bingbot", "OAI-SearchBot", "ChatGPT-User", "GPTBot",
        "Claude-SearchBot", "Claude-User", "ClaudeBot", "PerplexityBot", "Perplexity-User",
        "Applebot", "Applebot-Extended", "CCBot"]

def parse(text):
    groups, cur, last_ua = [], None, False
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        m = re.match(r"(?i)^(user-agent|allow|disallow)\s*:\s*(.*)$", line)
        if not m: continue
        field, value = m.group(1).lower(), m.group(2).strip()
        if field == "user-agent":
            if not last_ua: cur = {"agents": [], "rules": []}; groups.append(cur)
            cur["agents"].append(value.lower()); last_ua = True
        else:
            last_ua = False
            if cur is not None: cur["rules"].append((field == "allow", value))
    return groups

def rule_regex(path):
    end = path.endswith("$"); body = path[:-1] if end else path
    return re.compile("^" + ".*".join(re.escape(p) for p in body.split("*")) + ("$" if end else ""))

def verdict(groups, bot, path):
    named = [g for g in groups if bot.lower() in g["agents"]]
    chosen = named or [g for g in groups if "*" in g["agents"]]
    best = None  # (length, allow, rule)
    for g in chosen:
        for allow, rule in g["rules"]:
            if not rule: continue  # an empty Disallow allows everything
            if rule_regex(rule).match(path):
                key = (len(rule), allow)
                if best is None or key > best[:2]: best = (len(rule), allow, rule)
    group = "own group" if named else ("* group" if chosen else "no group")
    if best is None: return "allowed", group, "-"
    return ("allowed" if best[1] else "blocked"), group, ("Allow: " if best[1] else "Disallow: ") + best[2]

url = sys.argv[1]
u = urllib.parse.urlsplit(url)
path = (u.path or "/") + ("?" + u.query if u.query else "")
req = urllib.request.Request(f"{u.scheme}://{u.netloc}/robots.txt", headers={"User-Agent": "ai-robots-check/1.0"})
try:
    text = urllib.request.urlopen(req, timeout=20).read().decode("utf-8", "replace")
except urllib.error.HTTPError as e:
    if 400 <= e.code < 500 and e.code != 429: text = ""  # Google treats a 4xx (except 429) as no robots.txt
    else: sys.exit(f"robots.txt returned {e.code}: crawlers may treat the whole site as off limits for now")
groups = parse(text)
print(f"{u.netloc}{path}")
for bot in BOTS:
    v, g, r = verdict(groups, bot, path)
    print(f"  {bot:<18} {v:<8} {g:<10} {r}")

Run against two news sites on October 17, 2026, it returned:

www.nytimes.com/section/technology
  Googlebot          allowed  own group  -
  Google-Extended    blocked  own group  Disallow: /
  Bingbot            allowed  * group    -
  OAI-SearchBot      blocked  own group  Disallow: /
  ChatGPT-User       blocked  own group  Disallow: /
  GPTBot             blocked  own group  Disallow: /
  Claude-SearchBot   blocked  own group  Disallow: /
  Claude-User        blocked  own group  Disallow: /
  ClaudeBot          blocked  own group  Disallow: /
  PerplexityBot      blocked  own group  Disallow: /
  Perplexity-User    blocked  own group  Disallow: /
  Applebot           allowed  * group    -
  Applebot-Extended  blocked  own group  Disallow: /
  CCBot              blocked  own group  Disallow: /
www.theguardian.com/technology
  Googlebot          allowed  * group    -
  Google-Extended    allowed  * group    -
  Bingbot            allowed  * group    -
  OAI-SearchBot      allowed  * group    -
  ChatGPT-User       allowed  * group    -
  GPTBot             allowed  * group    -
  Claude-SearchBot   blocked  own group  Disallow: /
  Claude-User        blocked  own group  Disallow: /
  ClaudeBot          blocked  own group  Disallow: /
  PerplexityBot      blocked  own group  Disallow: /
  Perplexity-User    allowed  * group    -
  Applebot           allowed  * group    -
  Applebot-Extended  blocked  own group  Disallow: /
  CCBot              blocked  own group  Disallow: /

The Times' file has two User-agent: Googlebot groups; the script merges them, as Google does. Both publishers chose these rules on purpose. The point is how fast the differences show up: the Guardian's file treats OpenAI's and Anthropic's search bots differently, and it is easy to miss that by reading 60 lines by eye.

The script checks robots.txt only. It doesn't see firewalls, bot protection or login walls, which can block a crawler the file allows.

How do you read the results?

Read each row as "which group applied" and "which rule decided it", then compare with what you meant.

Two rows need judgement. A blocked result for ChatGPT-User or Perplexity-User doesn't guarantee those agents stay out, because their operators say robots.txt may not apply to user-requested fetches. And a blocked Google-Extended row has no effect on Google Search.

Which robots.txt mistakes block AI crawlers by accident?

Most accidental blocks come from group selection, not from the rules themselves.

• A named group that drops your * rules. Copy every Disallow you still need into the new group, as in our GPTBot test above.
• Testing with the wrong parser. For User-agent: *, Disallow: / then Allow: /blog/, Python 3.11's urllib.robotparser returned blocked for /blog/post, because it took the first matching rule. Google's longest-match rule allows it, and so does the script above.
• Blocking a search bot when you meant a training bot. OAI-SearchBot, Claude-SearchBot and PerplexityBot decide search visibility in their own products; GPTBot, ClaudeBot and CCBot are the training settings. Our directory of AI crawlers has a recipe that opts out of training and stays open to search.
• A platform default you didn't write. Squarespace's file lists 26 AI crawlers but only blocks them when you tick its AI checkbox; our run on its Bedford demo showed every bot allowed. See the Squarespace guide and, for WordPress, where WordPress keeps robots.txt.
• Expecting an instant change. Google, OpenAI and Perplexity each describe caching or delays of about a day.
• Using robots.txt to remove pages. robots.txt stops crawling, not indexing; a blocked URL Google already knows about can stay as indexed, though blocked by robots.txt. Use noindex on a crawlable page instead.

Where does FreshlyCited fit?

FreshlyCited works on the Google side of this check. "ChatGPT, Perplexity and other assistants use their own crawlers and search partners. We don't submit to them, and we don't claim to." The robots.txt rows for those assistants are settings only you control.

For Google, FreshlyCited checks whether each URL is indexed ("Seen by Google" or "Invisible"), pushes only the invisible ones and re-checks them on days 1, 3 and 7; its pre-flight check flags URLs that robots.txt blocks for Googlebot, so you can fix the rule instead of spending a credit. For sites you verify with an IndexNow key, changes are also sent to Bing. The free plan gives you 10 free links to try, then 100 free checks a month, with no card. Check my pages free.