How to Generate a Sitemap Automatically

A sitemap does not guarantee indexing. Its job is to tell search engines which important URLs exist, which pages changed, and which pages may be worth crawling again.

The sitemaps.org protocol also explains that sitemaps provide URL metadata to search engines, but do not guarantee inclusion in search results.

When your site has only a few pages, a hand-written sitemap.xml is fine. Once you have blog posts, templates, tools, integrations, city pages, product pages, or docs, the sitemap should be generated automatically.

This article references official materials:

Which Pages Belong in a Sitemap

A sitemap is not a dumping ground for every URL.

Include pages you want search engines to discover, crawl, and index.

Good candidates:

text

Homepage
Core product pages
Blog posts
Documentation pages
Template pages
Tool pages
Integration pages
Category pages
Important topic pages
Localized pages

Poor candidates:

text

Login pages
Admin pages
Payment callback URLs
Search result pages
Filtered parameter pages
Duplicate content pages
Drafts
Noindex pages
Private user pages
Temporary preview pages

Google's sitemap documentation also recommends including URLs you want to see in search results.

Every URL Needs a Source

The first step in automatic generation is defining URL sources.

Common sources:

text

Static routes: /, /pricing, /about
Blog database: /blog/[slug]
Docs system: /docs/[slug]
Template library: /templates/[slug]
Tool pages: /tools/[slug]
Integration pages: /integrations/[app-a]/[app-b]
Localized routes: /en/..., /zh/...

Do not hard-code a large list of URLs.

A better approach: each page type has a data source, and the sitemap collects only published, indexable pages from these sources.

Include Only Indexable Pages

Always filter by page state.

Each content item should have:

text

slug
status
noindex
updated_at
locale
canonical_url

Generation should filter:

text

status must be published
noindex must not be true
slug must be valid
canonical_url must not point to another page
page must not be private
page must not be a temporary preview

This is important. Otherwise drafts, tests, duplicates, and private pages may be submitted to search engines.

XML Fields

The most important sitemap field is loc.

The sitemaps.org protocol requires a loc child entry for each URL. Fields such as lastmod, changefreq, and priority are optional.

Common fields:

text

loc: full URL
lastmod: last modification time
changefreq: update frequency hint
priority: relative priority hint

Early on, focus on loc and lastmod.

Do not invent changefreq and priority values blindly. Search engines may not strictly use them, and wrong values do not create magic ranking improvements.

Generate With Next.js

If you use Next.js App Router, you can use app/sitemap.ts.

Next.js documentation explains that sitemap.(xml|js|ts) is a special file convention for generating a file that matches the sitemap XML format.

A simplified example:

import type { MetadataRoute } from 'next'

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://example.com'
  const posts = await getPublishedPosts()

  return [
    {
      url: baseUrl,
      lastModified: new Date(),
    },
    ...posts.map((post) => ({
      url: `${baseUrl}/blog/${post.slug}`,
      lastModified: post.updatedAt,
    })),
  ]
}

In a real project, do not hard-code baseUrl in many places. Use environment variables or a shared config.

Large Sites Need Sitemap Splitting

The sitemaps.org protocol limits a single sitemap to 50,000 URLs and 50MB uncompressed.

If your site grows beyond that, split sitemaps:

text

/sitemap.xml
/sitemap-posts-1.xml
/sitemap-posts-2.xml
/sitemap-templates.xml
/sitemap-integrations.xml

Then use a sitemap index to reference child sitemaps.

Early sites usually do not need this, but it is good to design with expansion in mind.

Localized Pages Need Care

If your site has Chinese, English, Japanese, or other localized pages, the sitemap should not include only one language.

Consider:

text

Does each language have its own URL?
Does canonical point to the page itself?
Do you need hreflang?
Is the translation published?
Are all localized pages indexable?

Do not include unfinished translations or raw machine translation drafts.

Common localization mistakes include wrong canonicals between languages and many empty translated pages in the sitemap.

When to Regenerate

A sitemap does not always need to query the database on every request.

Common strategies:

text

Build-time generation: good for static content
Scheduled generation: good for hourly or daily updates
On-demand revalidation: good for Next.js ISR
Background job generation: good for many pages
Cached output: good for high-traffic sites

If content changes rarely, build-time generation is enough.

If you have many user-generated, template, or generated pages, use a background job and store the result in object storage or cache.

Point robots.txt to the Sitemap

You can expose a sitemap in several ways:

text

Declare it in robots.txt
Submit it in Google Search Console
Expose /sitemap.xml on the site

In robots.txt:

text

Sitemap: https://example.com/sitemap.xml

This helps crawlers discover the sitemap when they read robots.txt.

Automated Checks

After generating a sitemap, check it.

Recommended checks:

text

/sitemap.xml is reachable
Content-Type is correct
URLs are absolute
Draft and noindex pages are excluded
lastmod is reasonable
URLs return 200
Redirect URLs are excluded
Local test domains are not included
50,000 URL and 50MB limits are respected
Search Console can read it

A sitemap is not a one-time file. It is SEO infrastructure that should evolve with your content system.

A Minimum Viable Setup

For an indie developer:

text

1. Add status, noindex, and updated_at to page data
2. Write a shared getIndexableUrls()
3. Generate sitemap with app/sitemap.ts or a build script
4. Output only published and indexable pages
5. Use absolute URLs with the production domain
6. Point robots.txt to /sitemap.xml
7. Submit it to Search Console
8. Check readability after each release

This setup is simple and avoids most sitemap mistakes.

Final Thoughts

The core of a sitemap is not XML. It is the rule for which pages deserve to be discovered by search engines.

Automatic generation depends on clear data sources, indexability state, update times, absolute URLs, localization rules, and ongoing checks.

Once those rules are clear, the sitemap becomes stable SEO infrastructure instead of a file you manually edit before each launch.

Next, we continue with content distribution: Do You Need RSS?.