"Indexed, though blocked by robots.txt": what it means and what to do

The warning matters because the indexed result is built without your content, so the listing usually has no description, and a page you meant to hide can still turn up in search and in the AI features built on Google's index. This guide quotes Google's definition, explains when to act, and covers the causes, how to confirm them, the three ways to fix them, and validation.

What does "Indexed, though blocked by robots.txt" mean?

"Indexed, though blocked by robots.txt" means the URL is in Google's index even though your robots.txt file stops Googlebot from fetching it. As of October 2026, Google's help page defines it as "The page was indexed despite being blocked by your website's robots.txt file. Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page" (Page indexing report help).

Google builds that index entry from what other pages say about the URL, not from the page itself. Its robots.txt guide says "the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results" (Google Search Central). That's why the result looks thin: "any snippet shown in Google Search results for the page will probably be very limited."

It sits under the report's warnings, not its errors. Google explains that warnings "don't prevent a page from being indexed, but they do reduce Google's ability to understand and index your pages." The status dates from a January 11, 2021 change, when blocked pages that got indexed began to be reported "as "indexed but blocked" (warning) instead of "submitted but blocked" (error)" (Google Search Central Blog).

It's the mirror image of two sibling statuses:

Is "Indexed, though blocked by robots.txt" a problem?

"Indexed, though blocked by robots.txt" is a problem only when the indexed URL is one you either want out of Google or want shown properly. For URLs that are blocked to save crawling, such as cart or filter pages, it is usually harmless. Shopify's help center says the message "is normal behavior and usually not a cause for concern" and, for its default blocks, "In most cases, no action is needed" (Shopify Help Center).

If you need to know which blocked URLs Google is actually showing across several sites before deciding, check those URLs free.

What causes "Indexed, though blocked by robots.txt"?

The cause is always a crawl block on a URL that Google can still discover, usually through links. Google's robots.txt guide puts it plainly: "A page that's disallowed in robots.txt can still be indexed if linked to from other sites" (Google Search Central).

Blocked pages that are linked everywhere

Pages linked from every template, such as cart, account, login, sign-up or internal search, are blocked by many sites and platforms, and those links are enough for Google to know the URL. On Shopify, filtered and sorted collection URLs and cart links are blocked by default, which is why store owners often see this warning. Our Shopify guide covers the platform's defaults.

Disallow used to remove pages

Adding Disallow to get a page out of Google does the opposite of what people expect: Google stops crawling, but keeps the URL it already knows. Google's help says to "remove the robots.txt block and use 'noindex'" instead. Our WordPress robots.txt guide shows how this mistake happens on WordPress sites.

A noindex tag hidden behind a block

Many sites put noindex on a page and also disallow it, expecting both to help. They cancel out. Google's noindex documentation says "If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule" (Block Search indexing with noindex). We checked a live example on October 17, 2026: Wikipedia's search result pages carry a noindex tag, and its robots.txt disallows the /w/ path they live on.

curl -s "https://en.wikipedia.org/w/index.php?search=robots+txt&fulltext=1" \
  | grep -o -i '<meta[^>]*name="robots"[^>]*>'
# <meta name="robots" content="noindex,nofollow,max-image-preview:standard">

curl -s https://en.wikipedia.org/robots.txt > robots.txt
./robots_main robots.txt Googlebot "https://en.wikipedia.org/w/index.php?search=robots+txt&fulltext=1"
# user-agent 'Googlebot' with URI '...': DISALLOWED

robots_main is Google's open-source robots.txt parser (google/robotstxt), which we built from source. Googlebot never fetches that page, so it never reads the tag. We didn't check whether Google has these URLs indexed; the point is that the noindex can't do its job while the block is there.

A block added after the page was indexed

If you block a page that Google already had, the URL can stay in the index for a long time, because Google can no longer crawl it to see that anything changed. Shopify's help notes that when a normal page such as a homepage shows this warning, "it usually means Google's data about your store is outdated." An accidental block, such as a staging file shipped to production, often shows up this way first; see why a page is blocked by robots.txt for the usual culprits.

How do you confirm it with URL Inspection?

Confirm this status by inspecting the URL in Search Console: the indexed result shows the URL is on Google, while "Crawl allowed?" shows the robots.txt block. Our URL Inspection tool guide explains each field.

1. Read "Crawl allowed?" It shows whether Google could crawl the page or was blocked by a robots.txt rule.
2. Don't trust "Indexing allowed?" on a blocked page. Google's help warns: "If your page is blocked by robots.txt (see Crawl allowed?), then Indexing allowed? will always be "Yes" because Google can't see and respect any noindex directives" (URL Inspection tool help).
3. Find the rule. Test the URL against your live robots.txt with Google's open-source parser, or with a live URL Inspection test.
4. Look at the search result. Search for the exact URL in Google. A listing with a URL but little or no description matches what Google says a blocked result looks like.

How do you fix "Indexed, though blocked by robots.txt"?

Fix this status by choosing what you want the URL to be, then removing the robots.txt block in both of the first two cases. Google's next steps say: "If you do want to block this page from Google Search, robots.txt is not the correct mechanism to avoid being indexed. To avoid being indexed, remove the robots.txt block and use 'noindex'" (Search Console Help).

Remove the URL from Google

1. Remove the Disallow rule that matches the URL. Keep other rules as they are.
2. Add noindex, either as a meta tag in the page's <head> or as an HTTP response header. For PDFs and other non-HTML files, use the header. Google's documented forms are:

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

3. Keep the URL crawlable for good. Google says "If indexing or serving rules must be followed, the URLs containing those rules cannot be disallowed from crawling" (Robots meta tag documentation). Putting the Disallow back later hides the noindex again.
4. Hide it sooner if needed. Google's removals guide says the Removals tool can remove a page "from Google's search results within a day", but requests "last for about 6 months" (Remove a page from Google), so do it alongside noindex, not instead of it.

Password protection or deleting the page are alternatives Google lists for content that must not be reachable at all. Our guide to Excluded by 'noindex' tag covers how to check the tag is really served.

Keep the URL in Google

Remove the rule that blocks it, check the live file, and request a recrawl of robots.txt in the robots.txt report. Once Google can crawl the page, it can index the real content and build a proper listing.

Leave it

For cart, checkout, sort and filter URLs that you block to save crawling, no action is needed. Google's help says blocked URLs are "probably intentionally blocked" and asks you to use your judgment.

How do you validate the fix?

Validate the fix by clicking Validate fix on the warning once all affected URLs are unblocked, then watching where they move. Google says validation "typically takes up to about two weeks, but in some cases can take much longer" (Search Console Help).

• Removed with noindex: the URLs should move to "Excluded by 'noindex' tag". Google's help says a page "marked noindex" counts as fixed and is shown in the "Other" validation state, which is not an error.
• Kept and unblocked: the URLs should move to "Indexed", or to another status such as crawled, currently not indexed if Google decides not to keep the content.
• Left as is: don't validate. Validation stops when Google finds one remaining instance.

When should you push the URL for a recrawl?

Push for a recrawl only for URLs you unblocked and want indexed. For URLs you want removed, Google needs to recrawl them to see the noindex, and its noindex guide says "Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page" (Block Search indexing with noindex); the Removals tool covers the gap.

• A few important URLs: use Request indexing after the live test passes. Google says "there's a quota for submitting individual URLs" (Ask Google to recrawl).
• Many URLs: resubmit a sitemap that lists only pages you want indexed.

FreshlyCited helps on both sides. Its index check shows whether each URL is "Seen by Google" or "Invisible", so you can confirm noindexed URLs have dropped out and unblocked ones are in. For unblocked pages Google still can't see, smart push sends only the invisible ones and re-checks on days 1, 3 and 7; the pre-flight check flags any URL still blocked or set to noindex so you can skip it free. Google still decides what it indexes.

Sorting out blocked pages? Check which blocked URLs Google has: 10 free links to try, then 100 free checks a month, no card needed.