As Bronitoring already supports Sitemap XML Analyzer, it would be great to have another tool, but for the opposite purpose – to create or generate a sitemap.xml for a given website. Let’s dive in!

What Is a Sitemap?

Let me quote Google:

A sitemap is a file where you provide information about the pages, videos, and other files on your site, and the relationships between them. Search engines like Google read this file to crawl your site more efficiently.

A sitemap tells search engines which pages and files you think are important in your site, and also provides valuable information about these files. For example, when the page was last updated and any alternate language versions of the page.

https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview

Do We Really Need Sitemap.xml?

Actually, you don’t for some reasons.

For clarification purposes, let me cite Google once again; I like their explanation.

If your site’s pages are properly linked, Google can usually discover most of your site. Proper linking means that all pages that you deem important can be reached through some form of navigation, be that your site’s menu or links that you placed on pages. Even so, a sitemap can improve the crawling of larger or more complex sites, or more specialized files.

You might need a sitemap if

• Your site is large. Generally, on large sites it’s more difficult to make sure that every page is linked by at least one other page on the site. As a result, it’s more likely Googlebot might not discover some of your new pages.
• Your site is new and has few external links to it. Googlebot and other web crawlers crawl the web by following links from one page to another. As a result, Googlebot might not discover your pages if no other sites link to them.

You might not need a sitemap if

• Your site is “small”. By small, we mean about 500 pages or fewer on your site. Only pages that you think need to be in search results count toward this total.
• Your site is comprehensively linked internally. This means that Googlebot can find all the important pages on your site by following links starting from the home page.

Who Are Potential Users Of Such A Generator?

Software and web developers, who prefer using Next.js (or any popular web development framework, such as Ruby on Rails, Django, etc.), rather than WordPress or other CMS for their platforms or websites.

Many of these CMS already have internal support for generating sitemaps from existing content, but for those who don’t have such functionality, Bronitoring Sitemap XML Generator will help solve this task.

For example, Bronitoring uses Next.js for both parts: frontend and backend. Also, I use it for the landing page and other internal pages because I don’t want to have additional CMS for the website at the moment of writing.

I know there are many plugins for Next.js as well as for any TS project, but I don’t want to extend my codebase for such a tiny specific task, as nobody wants to. So, my main goal is to have a web tool that can provide me with automatically generated sitemap.xml from all the content available on my platform or website.

Yes, I will need to use such a tool every time when everything changes on my Bronitoring tool, such as a new tool, new internal page, you name it. I don’t want to manage this sitemap.xml manually and keep everything on my own, like valid UTF-8 encoding, encoded URLs, and last modification date.

Who Are My Competitors

At the moment of writing this article, I get only the first 10 results from Google SERP, because it’s enough to analyze their functionality and test out everything they have.

Here is the list of them, only 6 useful tools:

• https://www.xml-sitemaps.com/
• https://www.mysitemapgenerator.com/
• https://www.seoptimer.com/sitemap-generator
• https://www.powermapper.com/products/mapper/maps/sitemap-generator/
• https://smallseotools.com/xml-sitemap-generator/
• https://slickplan.com/xml-sitemap-generator

How Sitemap XML Should Work

In my mind, it could be as easy to use as possible for the users, like me. I don’t want to put all my internal URLs and pages neither in one single textarea, nor in a table with a bunch of textboxes.

To achieve that, I need to implement a full website crawler that analyzes the entire website and grabs all the links for its pages, providing us a sitemap with all pages found on the given website.

As I already have a full site scanner, but it’s not integrated into a web version, I can’t use such a tool for the first version of the Sitemap Generator. So, I would stick with a semi-automated generator first.

My Bronitoring tool has only 8 internal pages and I can easily put these links into the generator to get a sitemap as a result. It will be enough because I don’t want to spend so much time on such a small tool.

When I integrate the full-site scanner, then it will be much easier because the only thing you need is to enter your domain name and wait for 5-10 minutes to get everything from your website in one single place.

Which Attributes Will It Support

Regarding Google policy and Sitemap specification, the only required attribute is <loc>, as it states, “The location URI of a document”.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/foo.html</loc>
  </url>
</urlset>

Other attributes such as <lastmod>, <priority>, and <changefreq> can be skipped.

What Is The Next Step?

I want to build a first version of such a generator in a matter of days, where you can enter a list of your URLs from your website with additional optional attributes. The tool will analyze all the URLs for a 200 HTTP status code, encode given URLs in an appropriate format, and if it’s possible, it will get the last modification date. That’s it.

Also, you will be able to download the entire sitemap.xml or request this file via email (I’m not sure if I should do that in the first version, maybe I will skip this step).

That’s it.