Skip to content

Complete guideGoogle indexing & pages not indexed · 18 chapters

Google indexing: how it works and how to get pages indexed faster

Google indexing is the process where Google discovers a URL, crawls it, renders it and decides whether to store it in its search index. Only indexed pages can appear in Google Search. Google says crawling alone "can take anywhere from a few days to a few weeks," and it doesn't guarantee that any page will be indexed.

The story · 3 steps
Check my pages freeFree plan: 10 free links to try (once you verify your email), then 100 free checks a month, no card.

In this series

The guide, chapter by chapter

  1. Alternate page with proper canonical tag: is it a problem?

    What Alternate page with proper canonical tag means in Search Console, why it is usually fine, and how to find, fix and validate a wrong canonical.Read chapter 1
  2. "Blocked by robots.txt" in Search Console: what it means and how to fix it

    What "Blocked by robots.txt" means, when the block is intended, the rules that catch too much, and how to confirm, fix, validate and recrawl.Read chapter 2
  3. "Blocked due to other 4xx issue": what it means and how to fix it

    What "Blocked due to other 4xx issue" means in Search Console, which 4xx codes land there, the firewall and URL causes, and how to fix them.Read chapter 3
  4. Pages waiting on a bench outside the index library doors, with crawler footprints showing they were visited

    'Crawled - currently not indexed': what it means and how to fix it

    Google crawled your page but didn't index it. What this Search Console status really means, the common causes, and step-by-step fixes that work.Read chapter 4
  5. A long queue of pages waiting outside a crawler station beside a gold hourglass

    'Discovered - currently not indexed': why Google hasn't crawled your page

    Google knows the URL exists but hasn't crawled it yet. Why that happens, how crawl budget and server speed play in, and how to get the page crawled.Read chapter 5
  6. Duplicate without user-selected canonical: what it means and how to fix it

    What Duplicate without user-selected canonical means in Search Console, when to ignore it, the real causes, and how to fix and validate the status.Read chapter 6
  7. Duplicate, Google chose different canonical than user: why and how to fix it

    Why Google overrides your canonical tag, when to accept its choice, the conflicting signals behind it, and how to align, validate and recrawl.Read chapter 7
  8. Excluded by 'noindex' tag: find where noindex comes from and remove it

    What "Excluded by 'noindex' tag" means, the CMS, plugin, header and staging settings that add noindex, and how to remove it and validate it.Read chapter 8
  9. Google Indexing API: what it's for, quotas and setup

    What the Google Indexing API is for (job and livestream pages only), its 200-a-day quota, URL_UPDATED and URL_DELETED, setup, what to use instead.Read chapter 9
  10. A page on a tower ringing a gold bell as a crawler drone turns toward it

    How to get Google to crawl your site and recrawl updated pages

    Ask Google to recrawl updated pages, speed up crawling of new ones and manage crawl budget, so search engines pick up your latest version sooner.Read chapter 10
  11. A mechanical arm placing a page into a gold tube that leads to the search index building

    How to submit a URL to Google: 5 ways, from one page to thousands

    Submit a page to Google with URL Inspection, sitemaps or bulk tools: what each method does, its limits, and how to check the URL was indexed.Read chapter 11
  12. "Indexed, though blocked by robots.txt": what it means and what to do

    Why Google indexes URLs your robots.txt blocks, when to leave the warning, and the fixes: remove with noindex, keep by unblocking, or ignore.Read chapter 12
  13. A report with rows of coloured status bars and a magnifying lens over one row

    Search Console page indexing report: every status explained

    What each reason in Google Search Console's page indexing report means, which ones to ignore, which to fix, and how to confirm Google saw the fix.Read chapter 13
  14. "Sitemap could not be read" and "Couldn't fetch": causes and fixes

    Why Search Console says "Sitemap could not be read" or "Couldn't fetch", how to confirm the cause, validate the XML from a terminal and get it read.Read chapter 14
  15. Soft 404 in Google Search Console: what it means and how to fix it

    What a soft 404 means in Search Console, when to leave it, the real causes (empty pages, SPAs, rendering), and how to fix, validate and recrawl.Read chapter 15
  16. Google Search Console URL Inspection tool: a complete guide

    How Search Console's URL Inspection tool works: indexed result vs live test, rendered HTML, canonicals, Request indexing limits and its API.Read chapter 16
  17. A website on a small island hidden in fog while a lighthouse beam sweeps toward it

    Why is my website not showing up on Google? Causes and fixes

    Site not showing up on Google? Work through the causes in order, from noindex tags and robots.txt to crawl delays, and confirm each fix works.Read chapter 17
On this page · 9 sections
  1. 01How Google indexing works: discovery, crawling, rendering, indexing
  2. 02Why Google doesn't index everything, or not quickly
  3. 03How long indexing takes and what affects it
  4. 04How to check if a page is indexed
  5. 05How to get a new page indexed
  6. 06How to get an updated page re-crawled
  7. 07Common 'not indexed' statuses and what they mean
  8. 08Why indexing now decides eligibility for Google's AI Overviews and AI Mode
  9. 09Every guide in this series
A crawler drone following link threads between pages and carrying one into a grand library that stands for the search index

You can't force the timeline, but you can remove blockers, send clear signals and check what Google actually saw. It's worth the effort: Google still sends about 9 in 10 of the visits that search engines send to websites worldwide (StatCounter, September 2026), and its AI Overviews and AI Mode are built on the same index. That makes indexing in Google the first step for both classic rankings and Google's AI answers. This guide covers how search engine indexing works, why pages stall, how long it takes, and how to get new and updated pages indexed.

How Google indexing works: discovery, crawling, rendering, indexing

Google indexing works in four stages: Google finds a URL, fetches it, renders it like a browser, then decides whether the page is worth adding to its index. Google describes the same pipeline in How Search works.

Four stages on a conveyor: a page discovered, fetched by a crawler, assembled, and filed on an index shelf

Stage

What happens

Where it commonly fails

Discovery

Google learns the URL exists, from links on pages it already knows, from sitemaps or from a submission

Orphan pages with no internal links; URLs missing from the sitemap

Crawling

Googlebot requests the URL and downloads the response

robots.txt blocks, server errors, slow responses, crawl postponed

Rendering

Google runs the page's JavaScript to see the final content

Content that only appears after user interaction or blocked resources

Indexing

Google analyzes the content, picks a canonical and decides whether to store it

noindex, duplicates, thin or low-value content

Two points are easy to miss. First, discovery is not crawling: Google can know a URL for weeks without fetching it. Second, crawling is not indexing: Google can fetch a page, read it and still decide not to keep it. The Search Console statuses "Discovered - currently not indexed" and "Crawled - currently not indexed" map exactly to those two gaps.

Why Google doesn't index everything, or not quickly

Google doesn't index everything because it has limited crawl capacity and a quality bar. Its systems choose which URLs to crawl, how often, and which crawled pages are worth storing.

Google's crawl budget guide breaks crawling into two parts:

  • Crawl capacity limit: how much crawling your server can handle. If the site slows down or returns errors, "the limit goes down and Google crawls less."
  • Crawl demand: how much Google wants to crawl. It depends on perceived inventory, popularity ("URLs that are more popular on the Internet tend to be crawled more often") and staleness.

Then comes the indexing decision. Common reasons a crawled page isn't stored:

  1. It's a duplicate or near-duplicate, and Google indexed a different URL as the canonical.
  2. It has little unique value compared with pages Google already holds.
  3. It carries a noindex directive, sometimes left over from staging.
  4. It returns an error or a "soft 404," a page that says "not found" but returns a 200 status.

On top of that, the web keeps growing, much of it AI-assisted content. Google rations attention, so a new page on a young or weakly linked site waits longer than a new page on a site Google already crawls often.

How long indexing takes and what affects it

How long it takes Google to index a page ranges from about a day to several weeks, and Google publishes no fixed timeline. Its URL Inspection help says indexing "typically takes only a day or so, but can take much longer in some cases" (URL Inspection tool). Its recrawl guide says crawling "can take anywhere from a few days to a few weeks," and that "requesting a recrawl multiple times for the same URL won't get it crawled any faster" (Ask Google to recrawl your URLs).

What moves the needle:

Factor

Faster

Slower

Site crawl frequency

Established site Google visits daily

New domain with few links

Internal links

Linked from the homepage or a frequently crawled hub

Orphan page, buried deep

External links

Linked from pages Google already crawls and indexes

No links, or links on unindexed pages

Server health

Fast, stable responses

Slow responses, 5xx errors, timeouts

Page value

Unique, useful content

Thin, duplicate or templated content

Signals

Listed in the sitemap with an accurate lastmod

Missing from the sitemap, or lastmod that changes without real edits

Treat any tool or vendor promising instant Google indexing with caution. Google's own help says "Indexing is never instant, even when you submit a crawl request directly" (Page indexing report help).

How to check if a page is indexed

To check whether a page is indexed, use URL Inspection in Search Console for pages you own, and an index checker for many URLs or pages on other sites. A site: search gives a rough look but isn't proof.

  1. URL Inspection (Search Console help) shows whether the URL is on Google, the last crawl date, the canonical Google selected and any indexing issue. It only works on verified properties, one URL at a time.
  2. The Page indexing report in Search Console shows indexed and not-indexed counts for the whole property, grouped by reason. See the page indexing report explained.
  3. site: search. Google's own site: operator docs say the list "is not always exhaustive" and that an indexed URL can show up "however it's not guaranteed."
  4. Bulk index checkers query Google for each URL and record the answer. They're the practical option for hundreds of URLs, client sites or backlinks on sites you don't control. Our guide covers how to check if pages are indexed with each method.

Record the result with a date. Indexing changes over time, and a single check is a snapshot, not a verdict.

How to get a new page indexed

To get a new page indexed, make sure Google is allowed to index it, link to it from pages Google already crawls, list it in your sitemap, then request a crawl. Do it in that order, because a request for a blocked page wastes the request.

  1. Confirm the page is indexable. No noindex meta tag or X-Robots-Tag header (Block indexing with noindex), not disallowed in robots.txt, returns a 200 status, and the canonical points to itself.
  2. Make it worth indexing. Unique content that answers a real query. A thin or duplicate page will be crawled and dropped.
  3. Link to it internally from your homepage, a hub page or a recent post that Google crawls often.
  4. Add it to your XML sitemap with an accurate lastmod. Google says it uses lastmod "if it's consistently and verifiably" accurate (Build and submit a sitemap).
  5. Request indexing with URL Inspection. Search Console has a daily quota for individual submissions. For more detail on every option, see how to submit a URL to Google.
  6. Re-check after a few days. If it's still missing, look at its status in the Page indexing report before requesting again.

A note on the Google Indexing API: Google's documentation says it "can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject" (Using the Indexing API). It isn't a route for blog posts, product pages or backlinks, and Google says its spam policies apply to content submitted through it.

For many URLs at once, a push service sends Google a crawl signal per URL. The sensible version checks first and only acts on what's missing. FreshlyCited does exactly that: we check what Google has indexed, push the pages Google hasn't seen, then re-check on days 1, 3 and 7.

How to get an updated page re-crawled

To get an updated page re-crawled, change the page meaningfully, update its sitemap lastmod to the real edit time, and request a recrawl. Google keeps serving the version it last stored until it crawls again.

Updated pages are easy to forget. You refresh a pricing page, a comparison or a guide, and Google keeps showing the old title, old prices and old answers. Google's crawl demand includes staleness ("Our systems want to recrawl documents frequently enough to pick up any changes"), but popular pages get that attention first.

  • Update lastmod only for real changes. Google's sitemap docs say lastmod "should reflect the date and time of the last significant update," and a copyright-year change doesn't count.
  • Request indexing once in URL Inspection. Repeating it doesn't help.
  • Surface the update internally, for example a "recently updated" block on a crawled hub.
  • Don't fake freshness. Bumping dates without changing content is a pattern Google and Bing both discourage.

Our walkthrough shows how to get Google to recrawl your site, including crawl budget on large sites.

Common 'not indexed' statuses and what they mean

The Page indexing report in Search Console groups not-indexed pages by reason. Some reasons are normal and need no action; others mean a page you care about is invisible.

A sorting station diverting pages into different bins, with only one gold bin leading into the index

Status in the report

What it means

Usually needs action?

Discovered - currently not indexed

Google knows the URL but hasn't crawled it yet

Yes, if it lasts or affects many important pages

Crawled - currently not indexed

Google crawled it and chose not to index it, for now

Yes, if the page matters

Duplicate without user-selected canonical

Google sees it as a duplicate and you didn't declare a canonical

Sometimes

Duplicate, Google chose different canonical than user

Google ignored your canonical and picked another URL

Check it

Alternate page with proper canonical tag

Points correctly to an indexed canonical

No

Page with redirect

The URL redirects elsewhere

No, if intentional

URL marked 'noindex'

A noindex directive excludes it

Only if unintended

URL blocked by robots.txt

Googlebot isn't allowed to crawl it

Only if unintended

Soft 404

Looks like an error page but returns 200

Yes

Not found (404) / Server error (5xx)

The URL fails

Yes, for pages that should exist

The two statuses that cause the most confusion have their own guides: crawled - currently not indexed is usually a page-value problem, while discovered - currently not indexed is usually a crawl-priority problem. If your whole site seems absent, start with why your website isn't showing up on Google.

Google notes the report's example list "is limited to 1,000 items" and "isn't guaranteed to show all URLs in a given status" (Page indexing report help), so for large sites, check the URLs you care about directly.

Why indexing now decides eligibility for Google's AI Overviews and AI Mode

Google indexing now decides whether a page can appear in Google's AI answers, because AI Overviews and AI Mode link only to indexed pages. Google's rule is simple: to appear as a link in AI Overviews or AI Mode, a page must be "indexed and eligible to be shown in Google Search with a snippet" (AI features and your website). Being indexed is the first step. Whether an AI answer cites your page still depends on quality, relevance and authority.

Google also says there are "no additional technical requirements" for AI features, and that "indexing and serving isn't guaranteed." In other words, there is no separate AI submission channel. The page Google stored is the page AI Overviews and AI Mode can draw on, so an update that hasn't been recrawled can't be reflected there yet. Google says Gemini can ground its answers in "content from the Google Search index" too (Google's common crawlers).

Bing is a free extra. Microsoft says Copilot's web answers use the Bing search service (Microsoft Learn). So for sites you verify by hosting a key file, FreshlyCited also sends your updated pages to Bing through IndexNow (an open "this page changed" notice that Bing and other participating engines accept; Google doesn't use it), at no extra credit cost. ChatGPT, Perplexity and other assistants use their own crawlers and search partners. We don't submit to them, and we don't claim to.

Indexing is the first step, not the last. Our guide to AI search optimization covers what happens after the page is in the index.

Every guide in this series

Each guide in this series takes one Google indexing problem and walks through what it means, how to diagnose it and how to fix it.

Guide

What it covers

The page indexing report explained

Every status in Search Console's Page indexing report, which to ignore and which to fix

Why your website isn't showing up on Google

A cause-by-cause checklist, from noindex tags to manual actions

Crawled - currently not indexed

Why Google crawled a page and passed on it, and how to make it worth keeping

Discovered - currently not indexed

Why Google postponed the crawl, crawl budget and server capacity

Submit a URL to Google

Five ways to submit a page, from URL Inspection to bulk pushes, and their limits

Get Google to recrawl your site

Getting new and updated pages crawled, with sitemaps, internal links and recrawl requests

If you manage more pages than URL Inspection can handle, or links on sites you don't own, a routine helps: check every URL, push only the invisible ones, then re-check and keep a record. Check your own links free.

Rakesh Patel

Written by

Founder & CEO, Space-O Technologies · Founder, TextBolt

Rakesh Patel founded Space-O Technologies in 2010 and leads it as CEO, with 25 years of leadership experience; he has overseen the delivery of more than 3,000 mobile app projects. He is also the founder of TextBolt, a business email-to-SMS service, and the author of "Enterprise Mobility Strategy and Solutions" and "A Guide to Open311". Space-O built FreshlyCited to solve the "shipped but not seen" problem on its own and its clients' sites: pages and links that are live but that Google hasn't looked at yet.

Founded Space-O Technologies in 2010 · 25 years of leadership experience · author of 'Enterprise Mobility Strategy and Solutions' and 'A Guide to Open311'

Written for FreshlyCited, which helps Google and Bing find new and updated pages sooner. About FreshlyCited

FAQ

Questions people ask

How long does it take Google to index a new page?

Usually somewhere between a day and a few weeks. Google says indexing typically takes a day or so but can take much longer, and that crawling can take a few days to a few weeks. It depends on how often Google crawls your site, how the page is linked and whether Google thinks it's worth indexing.

Does submitting a URL guarantee indexing?

No. Submitting asks Google to crawl; Google still decides whether to index. If a page keeps getting crawled but not indexed, the fix is usually on the page itself.

Why does Google indexing matter for AI search?

Google's AI Overviews and AI Mode can only show a page as a supporting link if it's indexed and eligible to appear in Google Search with a snippet. Being indexed makes a page eligible, not cited: whether an AI answer uses it still depends on quality, relevance and authority.

Check my pages free.

See which of your pages Google has indexed, free.