SEO Sitemap

Sitemap & robots.txt

Verify a site's sitemap and robots.txt configuration.

Procedure

1. Run the sitemap check:

   python -m seo_kit.crawler.sitemap "<url>"
   

   or from Python:

   from seo_kit.crawler.sitemap import run_sitemap_check
   result = run_sitemap_check("<url>")
   print(result.to_dict())
   

2. Report:

   • robots.txt HTTP status (missing = 404, error, or 200 OK)
   • Whether a Sitemap: directive is declared
   • Whether the declared sitemap parses, and how many URLs it lists
   • A score (0-100) with findings

3. Recommend fixes:

   • Missing robots.txt → provide a minimal User-agent: *\nAllow: / + Sitemap: line.
   • Missing sitemap directive → add Sitemap: https://site.com/sitemap.xml.
   • Empty/small sitemap → ensure canonical pages are listed.

Notes

• Handles both sitemap indexes and flat urlset sitemaps.
• Requires network access to the target domain.
• Inspired by AgriciDaniel/claude-seo and seranking/seo-skills (both MIT).