Sitemap Audit

Sitemap Audit

Audit a website's XML sitemaps and produce an actionable SEO report.

When to use this skill

Use this skill when the user asks to:

• Audit, validate, or check an XML sitemap or sitemap index
• Find sitemap URLs that 404, redirect, are noindexed, or are blocked by robots.txt
• Find URLs in the sitemap that canonicalize elsewhere or sit on the wrong host/protocol
• Check that sitemaps are declared in robots.txt and stay under the 50,000-URL / 50MB limits
• Clean up lastmod, priority, and changefreq metadata
• Improve the quality and trustworthiness of the URL set submitted to search engines

Inputs to collect

Before starting, confirm with the user:

1. Site URL or sitemap URL — a live site root (e.g., https://example.com, which auto-discovers sitemaps via robots.txt and common paths) or a specific sitemap URL (--sitemap).
2. Scope — how many listed URLs to probe (default 2000). Large sitemaps can be sampled with --max-urls.
3. Probing — whether to fetch each listed URL to check status/canonical/noindex (default on). Use --no-probe for a fast structural-only pass (parse errors, size limits, lastmod hygiene).

Note on the indexable contract: every URL in an XML sitemap should be a final, canonical, indexable 200 page. A URL that redirects, 404s, is noindexed, is blocked by robots.txt, or canonicalizes elsewhere does not belong in the sitemap. Most checks here enforce that contract.

Workflow

Step 1: Discover, parse, and probe the sitemaps

Use scripts/collect_sitemap.py:

python3 scripts/collect_sitemap.py https://example.com --max-urls 2000 --output sitemap_inventory.json

Point directly at a sitemap with --sitemap:

python3 scripts/collect_sitemap.py https://example.com/sitemap_index.xml --sitemap --output sitemap_inventory.json

The script reads robots.txt for Sitemap: directives (falling back to /sitemap.xml and /sitemap_index.xml), follows sitemap index files, handles gzipped sitemaps, and records each sitemap's type, URL count, byte size, and any parse/fetch error. For every listed URL it captures the sitemap's lastmod/changefreq/priority, then probes the URL for status, redirect target, rel=canonical, meta robots, the X-Robots-Tag header, and whether robots.txt disallows it.

Step 2: Run the audit checks

Use scripts/audit_sitemap.py:

python3 scripts/audit_sitemap.py sitemap_inventory.json --output audit_report.json

Step 3: Evaluate the audit checks

Evaluate each check defined in references/audit-checks.md. The core checks are:

Step 4: Produce the report

Write a report following the structure in references/report-template.md:

1. Summary — sitemaps found, total URLs listed, URLs probed, issue counts by severity
2. Sitemap files — table of each sitemap: type, URL count, size, gzip, errors
3. Issues — one section per failing check, listing the URL, the problem, and the fix
4. Prioritized action list — ordered by severity and impact

Step 5: Recommend fixes

For each issue, give a concrete fix:

• Non-200, redirecting, noindex, robots-blocked, or canonicalized URLs: remove them from the sitemap (or, for redirects, replace each with its final destination URL). The sitemap should list only final indexable pages.
• Wrong host/protocol: rewrite URLs to the canonical host and https://.
• Size limits: split the sitemap into multiple files under 50,000 URLs / 50MB and reference them from a sitemap index.
• Not in robots.txt: add a Sitemap: line to robots.txt for each sitemap (or the index).
• lastmod hygiene: emit a valid W3C datetime reflecting the real last-modified date; drop priority/changefreq if they are arbitrary (Google ignores them).
• Base every recommendation on the observed data; do not invent URLs.

Optional: export the report

As a Word document (.docx)

If the user wants the report as a .docx (for example, to share with stakeholders or attach to a ticket), save the Markdown report to a file and convert it:

python3 scripts/md_to_docx.py report.md --output report.docx

scripts/md_to_docx.py uses only the Python standard library (no pip install) and renders headings, tables, lists, links, bold/italic, and code blocks. Offer this whenever a user asks for a Word doc, a .docx, or a shareable/downloadable report.

As a CSV of findings (.csv)

If the user wants the raw findings as a spreadsheet (for filtering, sorting, or triage in Sheets or Excel), convert the audit's audit_report.json directly:

python3 scripts/findings_to_csv.py audit_report.json --output findings.csv

scripts/findings_to_csv.py is also standard-library only. It writes one row per finding, with check and severity columns prepended and list fields (e.g. the pages sharing a duplicate value) joined with ; . Unlike the .docx, which reformats the written report, the CSV is a direct dump of the structured findings — offer it whenever a user wants the data itself, a spreadsheet, or to slice findings by check or severity.

Resources

• scripts/md_to_docx.py — convert the Markdown report into a Word (.docx) document (standard library only)
• scripts/findings_to_csv.py — flatten the audit findings JSON into a CSV, one row per finding (standard library only)
• references/audit-checks.md — full definitions, thresholds, and rationale for every audit check
• references/report-template.md — report output structure
• scripts/collect_sitemap.py — discover, parse, and probe a site's XML sitemaps
• scripts/audit_sitemap.py — run audit checks against a sitemap inventory JSON file