Sitemaps and robots.txt in Next.js: a working setup

Add app/sitemap.ts and app/robots.ts, list only indexable URLs with real dates, split large sitemaps, and keep staging out of Google. Next.js 16 code.

By Sohail Hussain21 November 20265 min read

In the Next.js App Router, a sitemap is a file at app/sitemap.ts that returns an array of URLs, and robots.txt is a file at app/robots.ts that returns crawl rules. Next.js serves them as /sitemap.xml and /robots.txt in the correct formats, so there is no plugin and no XML to write by hand. The work that matters is deciding which URLs go in, giving them honest dates, and making sure staging sites do not end up in Google.

The code below is for Next.js 16, current as of October 2026, and follows the same pattern as this site's own sitemap and robots files. Getting these two files right is the first part of the indexing work in our technical SEO audits, on any platform.

Step 1: How do you create a sitemap in Next.js?

Add app/sitemap.ts and export a default function:

ts
// app/sitemap.ts
import type { MetadataRoute } from "next";
import { getAllPosts } from "@/lib/posts";

const base = "https://www.example.com";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = (await getAllPosts()).filter((post) => !post.draft);

  return [
    { url: base },
    { url: `${base}/services` },
    { url: `${base}/about` },
    { url: `${base}/contact` },
    ...posts.map((post) => ({
      url: `${base}/blog/${post.slug}`,
      lastModified: post.updated ?? post.date,
    })),
  ];
}

Visit /sitemap.xml and you get a valid XML sitemap. The function can read from files, a CMS or a database. It is a special Route Handler that is cached by default, so it runs at build time unless it reads request data.

If you have a short, fixed list of pages, a static app/sitemap.xml file also works. Most sites outgrow it as soon as they add a blog.

Each entry can also carry an images array of image URLs, which Next.js writes as an image sitemap. It is worth adding for sites where image search brings visitors, such as product catalogues, property listings or portfolios. For a services site, the page URLs are enough, because Google finds images on the pages themselves.

For sites not built on Next.js, or to sanity-check what yours should contain, our XML sitemap generator turns a list of URLs into a valid file.

Step 2: Which URLs belong in the sitemap?

Only URLs you want in Google's index, in their canonical form, that return a 200 status. A sitemap is a list of recommendations. Including pages you do not want indexed sends mixed signals, and Search Console flags them.

Leave out:

  • Drafts and preview-only pages.
  • Pages with robots: { index: false }, such as thank-you pages.
  • URLs that redirect, including the non-canonical variant of a trailing-slash or www choice.
  • Filtered and sorted versions of listing pages.
  • Paginated pages beyond the first, unless they hold content found nowhere else.

For dates, use the page's real last-change date. Do not set lastModified: new Date() on every entry. That tells search engines every page changed on every build, and Google's sitemap guidance says it uses lastmod only when it is consistently accurate. Google ignores priority and changeFrequency, so you can leave them out.

XML sitemaps: what should be in yours covers these rules for any platform.

Step 3: What if the site has more than 50,000 URLs?

A single sitemap file is limited to 50,000 URLs or 50 MB uncompressed. Below that, one file is fine. Above it, split the sitemap with generateSitemaps:

ts
// app/products/sitemap.ts
import type { MetadataRoute } from "next";
import { countProducts, getProducts } from "@/lib/products";

const perFile = 50000;

export async function generateSitemaps() {
  const files = Math.ceil((await countProducts()) / perFile);
  return Array.from({ length: files }, (_, id) => ({ id }));
}

export default async function sitemap(props: { id: Promise<string> }): Promise<MetadataRoute.Sitemap> {
  const id = Number(await props.id);
  const products = await getProducts({ offset: id * perFile, limit: perFile });
  return products.map((p) => ({
    url: `https://www.example.com/products/${p.slug}`,
    lastModified: p.updatedAt,
  }));
}

The files are served at /products/sitemap/0.xml, /products/sitemap/1.xml and so on. Since Next.js 16 the id arrives as a promise of a string, so convert it before doing arithmetic. You can also put a separate sitemap.ts in each section folder, such as app/blog/sitemap.ts. List every file in robots.txt, which accepts an array of sitemap URLs, or submit each in Search Console.

For multilingual sites, each entry can carry alternates.languages with the URL of every language version, which Next.js writes as hreflang links inside the sitemap. Internationalised routing in Next.js shows where those URLs come from.

Step 4: How do you create robots.txt?

Add app/robots.ts:

ts
// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: "*", allow: "/", disallow: ["/api/", "/search"] },
    sitemap: "https://www.example.com/sitemap.xml",
  };
}

Keep the disallow list short. Robots.txt controls crawling, not indexing. A disallowed URL that other sites link to can still appear in Google, with no description, because Google cannot read the page to see a noindex. To keep a page out of search, let it be crawled and set robots: { index: false } in its metadata. Never disallow /_next/: Google needs the CSS and JavaScript to render your pages.

To check a rule before you ship it, paste the file and a URL into our robots.txt tester. Robots.txt: what to block and what not to goes through the common cases.

Step 5: How do you keep staging and previews out of Google?

Make robots.ts depend on the environment. On Vercel, VERCEL_ENV is production only for production deployments:

ts
// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  if (process.env.VERCEL_ENV !== "production") {
    return { rules: { userAgent: "*", disallow: "/" } };
  }
  return {
    rules: { userAgent: "*", allow: "/" },
    sitemap: "https://www.example.com/sitemap.xml",
  };
}

On other hosts, use your own environment variable set only in production. A disallow-all file stops crawling but, as above, does not guarantee a URL stays out of the index if it is linked from somewhere. For a staging site, password protection is the reliable answer. Do it before the first link to staging is pasted into an email or a project tool.

The reverse mistake is worse: a disallow-all robots.txt or a site-wide noindex that ships to production at launch. It is the first item on our technical SEO checklist for that reason.

Step 6: How do you submit the sitemap?

In Google Search Console, open Sitemaps, enter sitemap.xml and submit. The status should show "Success" and a count of discovered URLs that matches what you expect. Submit once; after that Google re-reads the file on its own, and the Sitemap line in robots.txt lets other search engines find it.

Then check the Pages report over the following weeks. If many sitemap URLs show as "Discovered, currently not indexed", the problem is usually the pages, not the sitemap.

Sitemaps and robots files are one part of the search setup Next.js handles in code. Next.js for SEO covers the rest, and why we build company websites with Next.js explains why we prefer this to a plugin.

Was this guide useful?

Questions

Frequently asked.

Should I use priority and changefreq in a Next.js sitemap?

You can, and Next.js supports both fields, but Google says it ignores them. Spend the effort on an accurate lastModified date instead, because Google does use that when it is consistently correct. Leaving both fields out also keeps the file shorter and easier to check by eye.

Does Next.js update the sitemap when I publish a new post?

It updates when the sitemap is regenerated. sitemap.ts is cached by default, so on a static site the new URL appears after the next build. If posts come from a CMS without rebuilds, read them in sitemap.ts through the same cached, tagged function the pages use, so revalidating that tag on publish refreshes the sitemap too.

Sohail Hussain

Founder, MSH Technologies

Sohail has designed and built websites, web apps and AI products for over 12 years, for clients from Singapore-based industrial groups to early-stage SaaS startups. He writes about what actually worked.

Keep reading

Related guides.

All posts

Your next move

Like what you see? Tell us about it.

A process that takes too long, a website that undersells you, or an idea ready to take shape. Tell us where you are and we will suggest a useful next step.

India · Working with clients worldwide

Sohail, Founder, replies within 1 working day (Mon–Sat, IST). Free and no obligation.