"Blocked by robots.txt" in Search Console: what it means and how to fix it

Google can only show a page in search, and in the AI features built on its index, after it has crawled it, so one wrong line can hide a whole section of a site. This guide quotes Google's definition, separates intended blocks from mistakes, and covers the real causes, how to confirm and fix them, how to validate, and when to ask for a recrawl.

What does "Blocked by robots.txt" mean?

"Blocked by robots.txt" means Googlebot found the URL but a Disallow rule in the robots.txt file for that host told it not to request the page. As of October 2026, Google's help page lists this reason as "URL blocked by robots.txt" and defines it as "This page was blocked by your site's robots.txt file" (Page indexing report help). The same page says pages blocked across a site "should be marked in the Page indexing report as blocked by robots.txt."

Google is clear about what robots.txt is for: it is "used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google" (Google Search Central: robots.txt introduction). Each file covers one protocol and host: "Robots.txt files are not inherited by subdomains or parent domains, and a given page can be affected by only one robots.txt file" (robots.txt report help).

You may also see "Submitted URL blocked by robots.txt" in older guides and screenshots. That was an error in the former Index Coverage report for blocked URLs listed in a sitemap. In its January 11, 2021 update, Google said blocked pages that got indexed would be reported "as "indexed but blocked" (warning) instead of "submitted but blocked" (error)" (Google Search Central Blog).

Three statuses involve crawl and index rules, and they need different fixes:

Is "Blocked by robots.txt" a problem?

"Blocked by robots.txt" is a problem only for URLs you want people to find in Google. The help page itself says "URLs blocked by robots.txt are probably intentionally blocked. Use your judgment when deciding whether to address a given issue" (Search Console Help).

Leave it alone when the listed URLs are:

• Admin, login, cart, checkout or account pages
• Internal search results and sort or filter URLs
• Platform defaults doing their job. Shopify's default file, for example, "keeps cart links from being crawled, because they're unique to each customer" and keeps sorted collection pages out "because they return the same products in a different order" (Shopify Help Center).

Treat it as a problem when:

• A product, service, article or landing page you want in search appears in the examples.
• The count jumps after a deploy or migration. Google's help says that if indexed pages drop "without a corresponding increase in errors, you might be blocking access to your existing pages via robots.txt".
• Your sitemap lists blocked URLs. Google names "a sitemap that includes URLs that are blocked for crawling by robots.txt" as a cause of error spikes.

If you look after many sites and want to know which important URLs Google has indexed before you start editing rules, check those URLs free.

What causes "Blocked by robots.txt"?

A rule in the robots.txt file that applies to Googlebot matches the URL's path. The rule is either intended, or it is matching more than its author meant. Google's crawlers pick "the most specific rule based on the length of the rule path", and "in case of conflicting rules, including those with wildcards, Google uses the least restrictive rule" (Google's robots.txt specification).

A short rule that matches more than intended

Rules match by prefix, so a short path catches every URL that starts with it. Google's spec says /fish "Matches any path that starts with /fish. Note that the matching is case-sensitive." We ran Google's open-source robots.txt parser (google/robotstxt, built from source on October 17, 2026) against our own live robots.txt, which contains Disallow: /check for an app page:

curl -s https://freshlycited.com/robots.txt > robots.txt
./robots_main robots.txt Googlebot https://freshlycited.com/checklist
# user-agent 'Googlebot' with URI 'https://freshlycited.com/checklist': DISALLOWED

We have no /checklist page, but if we published one, it would be blocked by a rule written for something else. Anchoring the rule fixes it; the same tool reported these results:

User-agent: *
Disallow: /check$
Disallow: /check/
Disallow: /check?

With that file, /check, /check/abc and /check?url=x were DISALLOWED and /checklist was ALLOWED.

A staging robots.txt shipped to production

A staging site usually carries User-agent: * and Disallow: /, which blocks every URL. When that file reaches production during a launch or migration, the count rises across the whole site at once. Our test of that two-line file returned DISALLOWED for /pricing, as expected.

A forgotten Googlebot-only group

Google follows only one group of rules per crawler: "the group with the most specific user agent that matches the crawler's user agent. Other groups are ignored" (robots.txt specification). A leftover User-agent: Googlebot group with Disallow: / blocks Google even if your * group allows everything. It works the other way too: with a * group disallowing /admin/ and a Googlebot group that didn't repeat it, our test reported /admin/login as ALLOWED for Googlebot and DISALLOWED for Bingbot.

Another layer editing the file

Googlebot reads whatever the server returns at /robots.txt, which isn't always the file you edited: a CMS may generate it, an old file may override a plugin, or a CDN may prepend rules. Our WordPress robots.txt guide covers the virtual file, plugin editors and Cloudflare's managed rules; on Shopify stores, rules live in the robots.txt.liquid theme template. If you added rules for AI bots, check they name specific crawlers rather than *; our AI crawlers directory lists the tokens.

How do you confirm it with URL Inspection?

Confirm this status by inspecting an example URL in Search Console and reading the "Crawl allowed?" field. Google's help says "If Crawl allowed? is "No," that means Google can't crawl the page because of a robots.txt rule" (URL Inspection tool help). Our URL Inspection tool guide walks through the rest of the report.

1. Open the report row and inspect an example URL. Read "Crawl allowed?" and "Last crawl" in the indexed result.
2. Click Test live URL. If the live test still shows the block, the current file blocks it. If it doesn't, Google is working from an older crawl and you can move on to validation.
3. Open Settings > robots.txt report. It shows the file Google fetched for each host and when. To see older copies, "click the file in the files list in the report, then click Versions" (robots.txt report help). This tells you whether a bad version was live when Google crawled.
4. Find the exact rule. Google's report help points developers to "Google's open source robots.txt library, which is also used in Google Search", the tool we used above. Test the URL exactly as listed, since paths are case-sensitive.

How do you fix "Blocked by robots.txt"?

Fix this status by changing or removing the rule that matches the URL, in the place that actually serves your robots.txt, and then telling Google the file has changed. Keep the edit narrow so pages you meant to block, such as internal search, stay blocked.

1. Decide what the URL should be. If it belongs in Google, unblock it. If it should stay out of search, robots.txt is the wrong tool: Google says to "block indexing with noindex or password-protect the page" (robots.txt introduction), and noindex only works on a crawlable page.
2. Find the rule and the group it sits in. Check the group Googlebot follows, not just the * group.
3. Narrow or remove it. Use $ to end a match, add a trailing slash to limit a rule to a folder, or add a more specific Allow line. The longer path wins.
4. Edit through the right layer. On hosted platforms, use the platform's setting. Google notes that with a CMS such as Wix or Blogger "you might not need to (or be able to) edit your robots.txt file directly." On Shopify, the help center warns that editing robots.txt.liquid "is an unsupported customization" and that "Incorrect use of the feature can result in loss of all traffic."
5. Check the live file in a private window and test the URL again with the parser or a live URL Inspection test.
6. Ask Google to refetch robots.txt. In the robots.txt report, open the more settings menu next to the file and choose Request a recrawl. Google lists this case: "You changed your robots.txt rules to unblock some important URLs and want to let Google know quickly". It also notes that this doesn't mean the unblocked URLs themselves are recrawled straight away.
7. Remove blocked URLs from your sitemap if they are meant to stay blocked.

How do you validate the fix?

Validate the fix by clicking Validate fix on the "Blocked by robots.txt" reason once every URL you care about is unblocked. Google says validation "typically takes up to about two weeks, but in some cases can take much longer," and that an issue counts as fixed for a URL "When the URL is crawled and the issue is no longer found on the page" (Search Console Help).

• Validate only what you unblocked. If many URLs on the list are meant to stay blocked, validation will fail on them. Google's tip is to submit a sitemap of your important pages and filter the report by it first, since a smaller request "can complete faster".
• Expect a new status, not always "Indexed". Once crawled, a page can still land in crawled, currently not indexed if Google doesn't think it's worth keeping. That's a content question, not a robots.txt one.

When should you push the URL for a recrawl?

Push for a recrawl only after Google has fetched the corrected robots.txt and only for unblocked URLs you want indexed. Google caches robots.txt "for up to 24 hours" (robots.txt specification), and Request indexing won't work on a page the live test still sees as blocked: "You cannot request indexing if the page is considered to be non-indexable in the live test" (URL Inspection tool help).

• A few key pages: use Request indexing. Google says "there's a quota for submitting individual URLs and requesting a recrawl multiple times for the same URL won't get it crawled any faster" (Ask Google to recrawl).
• Many pages: resubmit a sitemap with accurate <lastmod> dates. Our guide to ways to submit a URL to Google compares the routes.
• Then re-check. Google says crawling "can take anywhere from a few days to a few weeks."

FreshlyCited fits at this step. Its pre-flight check flags URLs that are still blocked by robots.txt, so you can skip them free instead of spending a credit on a page Google can't crawl. It then checks which unblocked pages Google has indexed, pushes only the invisible ones, and re-checks them on days 1, 3 and 7. An index credit is used only if the link is indexed; if it still isn't after the day-7 re-check, the credit comes back automatically (exclusions apply, such as pages that return an error or carry noindex). Google still decides what it indexes.

Unblocked your pages? Check which URLs Google sees: 10 free links to try, then 100 free checks a month, no card needed.