Next.js sitemap, robots.txt and indexing: avoid the stale sitemap

A page Google hasn't indexed (added to its index, the library search results and AI answers are built from) can't show in either, and a stale sitemap is one of the quieter reasons new pages wait. This guide covers sitemaps, Search Console, Bing Webmaster Tools, IndexNow, noindex and the gotchas, using output from a Next.js 16.4.2 test app we built on October 17, 2026.

Where does a Next.js sitemap come from?

A Next.js sitemap comes from a sitemap.xml file or a sitemap.ts (or .js) file in the app directory; the code version exports a function that returns an array of URL objects. As of October 2026, the Next.js docs say "sitemap.js is a special Route Handler that is cached by default unless it uses a Request-time API or dynamic config option" (Next.js: sitemap.xml).

This is the file we built, with blog posts coming from a stand-in data function:

// app/sitemap.ts
import type { MetadataRoute } from 'next'
import { getPosts } from '@/lib/posts'

const BASE_URL = 'https://example.com'

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getPosts()
  return [
    { url: BASE_URL, lastModified: new Date(), changeFrequency: 'weekly', priority: 1 },
    ...posts.map((post) => ({
      url: `${BASE_URL}/blog/${post.slug}`,
      lastModified: post.updatedAt,
    })),
  ]
}

next build (Next.js 16.4.2, Cache Components off) wrote this to .next/server/app/sitemap.xml.body:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com</loc>
<lastmod>2026-10-17T14:56:05.300Z</lastmod>
<changefreq>weekly</changefreq>
<priority>1</priority>
</url>
<url>
<loc>https://example.com/blog/hello-world</loc>
<lastmod>2026-10-01</lastmod>
</url>
<url>
<loc>https://example.com/blog/pricing-update</loc>
<lastmod>2026-10-15</lastmod>
</url>
</urlset>

The Sitemap type accepts url, lastModified, changeFrequency, priority, alternates.languages, images and videos. Only some of them matter to Google, which says "Google ignores <priority> and <changefreq> values" and uses <lastmod> only if it is "consistently and verifiably (for example by comparing to the last modification of the page) accurate" (Google: build a sitemap). Use the date the content really changed, from your CMS or database, not new Date().

Splitting a large sitemap with generateSitemaps

Large Next.js sites split their sitemap with generateSitemaps, which returns a list of IDs; Next.js calls sitemap() once per ID. Google's limit is 50,000 URLs or 50MB uncompressed per sitemap file. Since Next.js 16, the function receives id as a promise that resolves to a string, so await it and convert it to a number (Next.js: generateSitemaps).

In our build, app/products/sitemap.ts with IDs 0 and 1 produced /products/sitemap/0.xml and /products/sitemap/1.xml. Two things we saw that the docs don't spell out: /products/sitemap.xml returned a 404, and the root /sitemap.xml didn't mention the product files. Nothing links them together, so tell search engines about each one (see the robots.txt section below).

How do you connect Search Console to a Next.js site?

Connect Google Search Console to a Next.js site by adding a property and verifying it with the HTML tag, which Next.js can write for you from metadata.verification. Put this in the root app/layout.tsx:

export const metadata: Metadata = {
  verification: {
    google: '<google-token>',
    other: { 'msvalidate.01': '<bing-token>' }, // Bing's meta tag name
  },
}

Our build rendered both tags in the home page <head>:

<meta name="google-site-verification" content="&lt;google-token&gt;"/>
<meta name="msvalidate.01" content="&lt;bing-token&gt;"/>

The placeholder brackets are escaped because they were literal text; a real token prints as-is. Google says the tag goes "into the <head> section in the HTML of your site's non-logged-in home page", and Search Console "will periodically check for the presence of this tag", so leave it in place. The HTML tag only verifies URL-prefix properties: DNS "is the only way to verify a Domain property" (Search Console Help: verify ownership).

Then open the Sitemaps report and submit https://example.com/sitemap.xml. Use the URL Inspection tool on one page to check what Google fetched.

How do you connect Bing Webmaster Tools to a Next.js site?

Connect Bing Webmaster Tools by importing your verified site from Search Console, or by verifying with Bing's own meta tag. As of October 2026, Bing's help says imported sites "will be added and get automatically verified", and that Bing then imports any new sitemap it finds in your Search Console account (Bing Webmaster Tools help).

To verify by meta tag instead, Bing's tag uses the name msvalidate.01 with a 32-character code in content, per Microsoft's verification guide (Microsoft Learn); the verification.other entry above writes it. Bing also accepts a BingSiteAuth.xml file at the site root, which you can drop into public/. Our Bing Webmaster Tools guide covers the reports you get afterwards.

How do you send IndexNow from Next.js?

Send IndexNow (an open "this page changed" notice that Bing and other participating engines accept; Google doesn't use it) from Next.js by hosting a key file and POSTing changed URLs after each deploy. Next.js serves anything in public/ from the site root, so public/<your-key>.txt lands at https://example.com/<your-key>.txt (Next.js: public folder).

IndexNow's documentation allows "up to 10,000 URLs per post", and a keyLocation field when the key isn't at the root (IndexNow documentation). A 200 response confirms receipt, not indexing. Our IndexNow guide covers the protocol and response codes.

This post-deploy script reads your live sitemap and sends URLs whose lastmod is on or after a date:

// scripts/indexnow-after-deploy.mjs
// SITE=https://example.com INDEXNOW_KEY=<your-key> SINCE=2026-10-15 node scripts/indexnow-after-deploy.mjs
const { SITE, INDEXNOW_KEY, SINCE, DRY_RUN, SITEMAP_URL } = process.env
const xml = await (await fetch(SITEMAP_URL ?? `${SITE}/sitemap.xml`)).text()
const since = new Date(SINCE)
const urls = [...xml.matchAll(/<url>\s*<loc>(.*?)<\/loc>(?:\s*<lastmod>(.*?)<\/lastmod>)?/g)]
  .filter(([, , lastmod]) => lastmod && new Date(lastmod) >= since)
  .map(([, loc]) => loc)

const body = {
  host: new URL(SITE).host,
  key: INDEXNOW_KEY,
  keyLocation: `${SITE}/${INDEXNOW_KEY}.txt`,
  urlList: urls.slice(0, 10000),
}
if (DRY_RUN) {
  console.log(JSON.stringify(body, null, 2))
} else if (urls.length) {
  const res = await fetch('https://api.indexnow.org/indexnow', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json; charset=utf-8' },
    body: JSON.stringify(body),
  })
  console.log(`IndexNow responded ${res.status} for ${urls.length} URLs`)
}

We ran it with DRY_RUN=1 and SINCE=2026-10-10 against our test build's sitemap. It selected the blog post updated October 15 and skipped the October 1 one, but it also picked the home page, because lastModified: new Date() makes the home page look changed on every build. Accurate dates keep both Google and IndexNow honest. You can run the same logic from a Route Handler (for example app/api/indexnow/route.ts, protected with a secret) if your CMS calls a webhook on publish. Our IndexNow setup docs show how FreshlyCited can send these for you once the site is verified.

How do robots.txt and noindex work in Next.js?

Next.js writes robots.txt from app/robots.ts and the robots meta tag from metadata.robots (or generateMetadata). This robots.ts lists all three sitemaps from our build:

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/', disallow: '/api/' },
    sitemap: [
      'https://example.com/sitemap.xml',
      'https://example.com/products/sitemap/0.xml',
      'https://example.com/products/sitemap/1.xml',
    ],
  }
}

Build output:

User-Agent: *
Allow: /
Disallow: /api/

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/products/sitemap/0.xml
Sitemap: https://example.com/products/sitemap/1.xml

To keep a page out of the index, set robots: { index: false, follow: true } in that page's metadata; our build rendered <meta name="robots" content="noindex, follow"/>. Don't disallow the same path in robots.txt. Google says "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file" (Google: block indexing with noindex). Our WordPress robots.txt guide explains the crawl-versus-index difference in more depth, and Excluded by 'noindex' tag covers the Search Console status a stray tag produces.

Which Next.js gotchas keep pages out of the index?

The Next.js gotchas that most often keep pages out of Google are a sitemap frozen at build time, a noindex inherited from a layout, and 404s that stream as 200. Each one is quiet: the site works for visitors.

The stale sitemap

A stale Next.js sitemap is one generated at build time and never regenerated. With Cache Components off, our sitemap above showed as ○ /sitemap.xml (static) in the build output. We started the server and requested it twice, five seconds apart; both responses carried x-nextjs-cache: HIT and the same <lastmod>2026-10-17T14:56:05.300Z</lastmod>, the moment of the build. A post published after that build wouldn't appear until the next deploy.

Two fixes, both of which we built:

The revalidate option is documented for Route Handlers in the Next.js caching guide (previous model). Check which mode you're in: the app create-next-app@16.4.2 generated for us set cacheComponents: true in next.config.ts. In that mode, the same sitemap with new Date() built as ƒ /sitemap.xml (rendered on every request); without new Date() it was static again. Read the route table after every build rather than assuming.

noindex leaking from a layout

Metadata merges shallowly: the docs say nested fields such as robots "defined in an earlier segment are overwritten by the last segment to define them" (Next.js: generateMetadata). A robots: { index: false } set in a staging or section layout applies to every page below it that doesn't set its own robots. Search for index: false before launch.

404s that return 200

Next.js normally returns a 404 with a noindex tag; our unmatched test URL got 404 and <meta name="robots" content="noindex"/>. But the docs warn that after streaming starts, "the status code remains 200", with the noindex tag injected instead (Next.js: not-found.js). Call notFound() before any <Suspense> boundary streams, or Search Console may list those URLs as a soft 404.

Metadata that streams after the HTML

Since Next.js 15.2, generateMetadata can stream: its tags are appended to <body> for bots that run JavaScript, while "HTML-limited bots" such as facebookexternalhit still get them in <head>. The docs say this has been "verified" for Googlebot. If another crawler you care about misses your titles, the htmlLimitedBots option in next.config.ts controls the list.

Client-only pages

A page that fetches its content in the browser behaves like a single-page app, with the same rendering delays. If parts of your Next.js app are client-rendered, our React SEO guide covers how Google handles them.

If you've shipped a batch of pages and aren't sure which ones Google has picked up, check them before you push anything.

What is the verify and push checklist for Next.js?

Run this after each release that adds or changes public pages:

1. Read the route table in the next build output: /sitemap.xml and /robots.txt should be listed, with a revalidate time if your content changes between deploys.
2. Open /sitemap.xml and /robots.txt on production. Check the new URLs are there, lastmod dates are real, and every split sitemap is listed.
3. View source on a new page. Confirm the <title>, description and canonical are in the HTML, and that there's no unexpected noindex.
4. Inspect one new URL in Search Console's URL Inspection tool and run a live test.
5. Resubmit the sitemap if its URL changed, in Search Console and Bing Webmaster Tools.
6. Send IndexNow for changed URLs from your deploy script, or let FreshlyCited do it for verified sites.
7. Check what Google has indexed a few days later. Pages that stay out may show as discovered, currently not indexed, which usually means Google hasn't crawled them yet.

FreshlyCited does step 7 and the push for you: it checks each URL against Google ("Seen by Google" or "Invisible"), pushes only the invisible ones, and re-checks on days 1, 3 and 7. For sites you verify with an IndexNow key, changes also go to Bing ("Sent to Bing"). Google still decides what it indexes. To run it straight from your sitemap, follow the recipe for pushing unindexed sitemap URLs; if a coding agent ships your pages, our AGENTS.md snippet tells it what to do after each deploy. More frameworks and site builders are in the platform guides.

Deployed something new? Check your Next.js pages free: 10 free links to try, then 100 free checks a month, no card needed.