Sitemaps: the homework I do before using a generator

If you're the SEO person on a team, sooner or later a developer will ask you something like this: "Can't we just put the site into a sitemap generator and be done?"

The honest answer is: yes, you can. But before you do, there's some homework. And that homework is where most of the thinking happens.

This article is about that homework, what I do, whatever kind of site it is, and, just as important, what I choose not to do.

What is a sitemap, really?

A sitemap is a file that lists the pages on your website that you would like Google to look at. Think of it as saying, "Here are my pages. Please come and browse."

Notice the word please. A sitemap is a hint, not a command. Google's own documentation says that submitting a sitemap doesn't guarantee Google will download it or use it to crawl your URLs. So a sitemap helps Google find your pages. It doesn't force anything.

Does every site build its sitemap the same way?

The file gets made differently depending on the kind of site you have.

A WordPress (or similar CMS) site. Most of the work is done for you. Google notes that platforms like WordPress, Wix and Blogger often make a sitemap available already, and plugins such as Yoast SEO or Rank Math can generate one too.

A small site. If your site has only a small number of pages, this is also easy. Google's guidance is that a sitemap with fewer than a few dozen URLs can reasonably be written by hand. Open a text editor, list the pages, done.

A site with thousands of pages. This is where it stops being a quick task. Google suggests generating the file from your website's own software or database rather than maintaining it by hand, and that's a conversation with your developers. It's also where the thinking matters most, because someone has to decide which of those thousands of URLs belong in the file at all.

Whichever way the file gets made, one thing stays the same: someone has to decide which URLs belong in it. That's the homework, and it applies to every site, whether a plugin, a developer or a text editor produces the file.

But can't we just use a sitemap generator?

Yes. But a generator can only list what you give it. It doesn't know which pages matter.

That's why the initial mapping comes first. Before any tool runs, I'd want answers to these questions:

• Which pages do I actually want people to find in Google?
• Is there one preferred (canonical) URL for each of those pages, or do several URLs show the same content?
• Are any of the URLs redirects, errors, or pages I've marked noindex? Those don't belong in the file.
• Are there pages that exist for the site to work but not for search, like login pages or filtered views?

Google's documentation says to include the URLs you want to see in search results, and that those are your canonical URLs. A sitemap is basically a statement of which pages you consider worth finding. A list full of redirects and dead pages makes that statement messy.

Is crawl budget the reason?

This is the part I wanted to check, because "don't waste crawl budget" is something marketers repeat a lot.

Google's crawl budget guide is an advanced guide, and it says it is intended mainly for very large sites (1 million+ pages that change about weekly), medium or larger sites (10,000+ pages) that change very rapidly (daily), and sites with many URLs marked "Discovered - currently not indexed" in Search Console. For sites outside that, Google says keeping your sitemap up to date and checking your index coverage regularly is adequate.

So is crawl budget a reason to do the homework? Maybe, for a very large site. For a smaller one, I'd say the better reason is simpler: a clean sitemap is easier to maintain, easier to debug, and easier to trust.

What about the last modified date?

The <lastmod> tag tells Google when a page last changed. Google says it uses this value if it is consistently and verifiably accurate, for example, when it can compare it with the page's actual last modification.

Accurate is the key word. Google also says the date should reflect a significant update, such as a change to the main content, the structured data or the links on the page. Changing the copyright year in the footer doesn't count.

This is where generators can quietly go wrong. Some set every page's date to the moment the file was generated, so every page looks like it changed today, whether it did or not. A tool can only print the date you give it. So where does that date come from? On a hardcoded site, it might be a date you enter yourself when you edit a page, or a field like "updated" in each page's own data. The tool just formats it.

So does the generator add it correctly? Only if you've decided where the truth lives first.

If you want to know more about crawl budget, read out this reference article: https://developers.google.com/crawling/docs/crawl-budget

What about frequency and priority?

The sitemap format also allows <changefreq> (how often a page changes) and <priority> (how important it is). Many guides still tell you to fill these in carefully.

Google's documentation says it ignores both. So if a generator adds them, it's harmless, but it's also doing nothing for Google. Time spent debating whether the blog should be a 0.8 or a 0.6 is time better spent making the <lastmod> dates honest.

What I do

When I get a new site, I start by understanding the nature of the business or blog. That tells me what the sitemap needs to carry.

If it's a bigger site, I create child sitemaps rather than putting everything into one core sitemap. I also keep a separate child sitemap for videos and images.

Why child sitemaps for a larger site?

There are a few reasons that I can back up with Google's own guidance:

• Size limits. One sitemap can hold at most 50,000 URLs or 50 MB uncompressed. A larger site has to be split, and a sitemap index file can point to all the child files.
• Tracking. You can submit several sitemaps to Google, and doing so lets you track the search performance of each individual sitemap in Search Console.
• Easier to read and fix. When pages, images and videos each have their own file, it's easier to see where a problem sits.

What I don’t do

I don't send a site straight into a sitemap generator.

First I check that the canonical tags are set properly and that no pages return a 404 error. Only after those checks do I add the URLs to a generator.

That order matters, because a generator only lists what it's given. It won't tell you that a page is an error, or that it points to a different canonical URL.

So where does a good sitemap really start?

The generator is the quick part. The slow part is everything before it: knowing the site, choosing the URLs worth listing, and checking that each one is canonical and not an error.

The tags follow the same logic. <lastmod> is only worth including if the date behind it is true, and Google says it ignores <priority> and <changefreq> anyway.

So if the tool takes two minutes and the homework takes the rest of the afternoon, which of the two is really building your sitemap?