On this page · 6 sections
A page that fails any of these can't be indexed (added to the library Google answers from), so it can't appear in Google Search or in AI Overviews and AI Mode, however good it is. This guide covers what makes a page indexable, how to check by hand, a script that checks every signal at once, what each result means and how to fix it, and what a raw-HTML check can miss.
What makes a page indexable?
A page is indexable when Google can fetch it, is allowed to crawl it, finds no rule telling it not to index it, and isn't pointed to a different URL as the preferred version. As of October 2026, these are the five checks and the Search Console status each failure usually produces:
Check | Indexable when | Fails when | Search Console usually shows |
|---|---|---|---|
HTTP status | The URL answers | 404, 410, other 4xx, 5xx, or a redirect | Blocked due to other 4xx issue, Not found (404), Page with redirect |
robots.txt | No rule disallows the path for Googlebot | A | |
Robots meta tag | No | | |
| No | | Excluded by 'noindex' tag |
Canonical | Points to the URL itself, or is absent | Points to another URL | Alternate page with proper canonical tag or Google chose a different canonical |
Sources: Google's robots meta tag documentation, noindex guide and canonicalization methods. Google says it supports two ways to set noindex: "as a <meta> tag and as an HTTP response header", and the header "can be used as an element of the HTTP header response for a given URL." For the difference between indexability and indexation, see our index checker guide.
How do you check a page for noindex by hand?
You check for noindex by reading the response headers and the HTML your server sends. Two curl commands cover both:
URL="https://example.com/page"
# 1. Status line and any X-Robots-Tag header
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "$URL" | grep -i -E '^HTTP|x-robots-tag'
# 2. Robots meta tags, including googlebot-specific ones
curl -s "$URL" | grep -o -i -E '<meta[^>]+name=.(robots|googlebot).[^>]*>'Read every result, not just the first: a page can carry more than one robots meta tag, plus a header, and Google combines them. Our run against a WordPress.org login URL on October 17, 2026 found x-robots-tag: noindex,follow in the header and two meta tags, one with max-image-preview:large and one with noindex,follow. Google's rule settles it: "In the case of conflicting robots rules, the more restrictive rule applies."
Also check robots.txt for the path, because a blocked page's noindex is never read. Google warns that if the page is blocked, "the crawler will never see the noindex rule, and the page can still appear in search results" (Google: block indexing).
If one of your key pages turns out to be noindexed, fix it first; then check which of your pages Google has indexed, free.
How do you check every indexability signal at once?
You check every signal at once with a script that fetches the URL without following redirects, reads the header and meta tags, finds the canonical, and applies Google's robots.txt rules for Googlebot. We wrote this one and ran it on October 17, 2026:
# indexable.py: is this URL indexable by Google? Checks status, robots.txt, X-Robots-Tag, meta robots and canonical.
# Usage: python3 indexable.py https://example.com/page
import re, sys, urllib.request, urllib.error
from urllib.parse import urlsplit, urljoin
UA = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
url = sys.argv[1]
class NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *a, **k): return None
opener = urllib.request.build_opener(NoRedirect)
def fetch(u):
try:
r = opener.open(urllib.request.Request(u, headers={"User-Agent": UA}), timeout=20)
return r.status, r.headers, r.read(2_000_000).decode("utf-8", "replace")
except urllib.error.HTTPError as e:
return e.code, e.headers, ""
def robots_verdict(u):
# Google's rules: the googlebot group if one exists, else "*"; longest matching rule wins; Allow wins ties.
p = urlsplit(u); path = (p.path or "/") + ("?" + p.query if p.query else "")
status, _, text = fetch(f"{p.scheme}://{p.netloc}/robots.txt")
if status >= 400: return f"no robots.txt rules apply (HTTP {status})"
groups, cur, last = [], None, False
for raw in text.splitlines():
m = re.match(r"(?i)^(user-agent|allow|disallow)\s*:\s*(.*)$", raw.split("#", 1)[0].strip())
if not m: continue
f, v = m.group(1).lower(), m.group(2).strip()
if f == "user-agent":
if not last: cur = {"a": [], "r": []}; groups.append(cur)
cur["a"].append(v.lower()); last = True
else:
last = False
if cur and v: cur["r"].append((f == "allow", v))
chosen = [g for g in groups if "googlebot" in g["a"]] or [g for g in groups if "*" in g["a"]]
best = None
for g in chosen:
for allow, rule in g["r"]:
body = rule[:-1] if rule.endswith("$") else rule
rx = "^" + ".*".join(re.escape(x) for x in body.split("*")) + ("$" if rule.endswith("$") else "")
if re.match(rx, path) and (best is None or (len(rule), allow) > best[:2]): best = (len(rule), allow, rule)
if not best or best[1]: return "allowed for Googlebot" + (f" (Allow: {best[2]})" if best else "")
return f"BLOCKED for Googlebot by Disallow: {best[2]}"
status, headers, html = fetch(url)
print(url)
print(" HTTP status: ", status, f"-> {headers.get('Location')}" if 300 <= status < 400 else "")
print(" robots.txt: ", robots_verdict(url))
xrt = [v for k, v in headers.items() if k.lower() == "x-robots-tag"]
print(" X-Robots-Tag: ", "; ".join(xrt) or "none")
metas = re.findall(r"<meta[^>]+name=[\"'](?:robots|googlebot)[\"'][^>]*>", html, re.I)
print(" meta robots: ", " | ".join(metas) or "none")
canon = re.search(r"<link[^>]+rel=[\"']canonical[\"'][^>]*href=[\"']([^\"']+)", html, re.I)
canon_url = urljoin(url, canon.group(1)) if canon else None
print(" canonical: ", canon_url or "none", "(points elsewhere)" if canon_url and canon_url.rstrip("/") != url.rstrip("/") else "")
directives = xrt + [re.sub(r".*content=[\"']([^\"']*).*", r"\1", m, flags=re.S) for m in metas]
noindex = any(d.strip().lower() in ("noindex", "none") for x in directives for d in re.split(r"[,:]", x))
if 300 <= status < 400: verdict = "redirects: Google indexes the target URL, not this one"
elif status != 200: verdict = f"not indexable: HTTP {status}"
elif "BLOCKED" in robots_verdict(url): verdict = "not crawlable: Google can't read the page (or any noindex on it)"
elif noindex: verdict = "not indexable: noindex"
elif canon_url and canon_url.rstrip("/") != url.rstrip("/"): verdict = "indexable, but the canonical points elsewhere, so Google will usually index that URL instead"
else: verdict = "indexable (Google still decides whether to index it)"
print(" verdict: ", verdict)Here's what it returned for six real URLs, each showing a different result:
https://freshlycited.com/about
HTTP status: 200
robots.txt: allowed for Googlebot (Allow: /)
X-Robots-Tag: none
meta robots: none
canonical: https://freshlycited.com/about
verdict: indexable (Google still decides whether to index it)
https://freshlycited.com/sign-up
HTTP status: 200
robots.txt: BLOCKED for Googlebot by Disallow: /sign-up
X-Robots-Tag: none
meta robots: <meta name="robots" content="noindex, nofollow"/>
canonical: https://freshlycited.com (points elsewhere)
verdict: not crawlable: Google can't read the page (or any noindex on it)
https://login.wordpress.org/?redirect_to=https%3A%2F%2Fmake.wordpress.org%2Fcore%2F
HTTP status: 200
robots.txt: allowed for Googlebot
X-Robots-Tag: noindex,follow
meta robots: <meta name='robots' content='max-image-preview:large' /> | <meta name="robots" content="noindex,follow" />
canonical: https://login.wordpress.org/ (points elsewhere)
verdict: not indexable: noindex
http://freshlycited.com/about
HTTP status: 308 -> https://freshlycited.com/about
verdict: redirects: Google indexes the target URL, not this one
https://www.squarespace.com/homepage-variation
HTTP status: 200
robots.txt: allowed for Googlebot
canonical: https://www.squarespace.com (points elsewhere)
verdict: indexable, but the canonical points elsewhere, so Google will usually index that URL instead
https://freshlycited.com/no-such-page-123
HTTP status: 404
verdict: not indexable: HTTP 404(We trimmed the unchanged lines from the last three results.) Our sign-up page fails on purpose: it's blocked and noindexed because it has no place in search. The second case is the one to watch for on your own pages, because the block means Google never reads the noindex.
What does each result mean, and how do you fix it?
Each result points to one fix, and the order matters: make the page crawlable before you rely on any tag inside it.
Result | What it means | What to do |
|---|---|---|
HTTP 404 or 410 | The page doesn't exist | Restore it, or redirect it to its replacement; see soft 404 if the page shows an error but answers 200 |
Other 4xx or 5xx | The server refused or failed | Check firewall and bot rules for Googlebot, and server errors |
3xx redirect | This URL hands off to another | Check the target instead; a permanent redirect is, in Google's words, "A strong signal that the target of the redirect should become canonical" |
robots.txt blocked | Googlebot may not fetch the page | Remove or narrow the rule; use noindex, not robots.txt, to keep a page out |
noindex (meta or header) | You asked Google not to index it | Remove the tag or header at its source: CMS setting, SEO plugin, server or CDN rule |
Canonical points elsewhere | You named another URL as the preferred version | Fine for duplicates; otherwise point the canonical at the page itself |
Indexable | Nothing on the page blocks indexing | Check whether Google has actually indexed it |
Platform settings are the usual source of a stray noindex. Our guides cover where it hides on WordPress, Wix, Squarespace and Webflow.
What can a raw-HTML check miss?
A raw-HTML check reads what the server sends, so it misses anything that changes after JavaScript runs or that depends on who's asking.
- Noindex added or removed by JavaScript. Google warns that when it sees
noindexin the original HTML, it "may skip rendering and JavaScript execution" (Google: JavaScript SEO basics), so a tag your app removes later can still keep the page out. Use the live test in the URL Inspection tool to see the rendered HTML Google gets. - Different answers for different visitors. A firewall, CDN rule or geo block can treat Googlebot differently from your script. Compare with URL Inspection's live test, which fetches as Google.
- Google's own choices. A crawlable, self-canonical page with no noindex can still be left out because Google judged it not worth keeping, which shows as crawled, currently not indexed.
That last point is why an indexability check is only half the job. The other half is checking whether Google has indexed the page, and for more than a handful of URLs a bulk index checker saves time.
Where does FreshlyCited fit?
FreshlyCited runs the same five checks before it pushes anything. Its pre-flight check looks at the HTTP status, redirects, robots.txt for Googlebot, noindex in the meta tag or X-Robots-Tag header, and a canonical that points elsewhere, and flags each URL that fails so you can fix or skip it before using a credit. It then checks the rest against Google's index ("Seen by Google" or "Invisible"), pushes only the invisible ones and re-checks them on days 1, 3 and 7. The free Site X-ray on our homepage runs the robots.txt and noindex checks on your newest pages, with no sign-up. The free plan gives you 10 free links to try, then 100 free checks a month, with no card. Check my pages free.
Written by
