Site Content Catalog
Build a content inventory for any domain — keyless. The script discovers pages via
robots.txt Sitemap directives, sitemap.xml (with sitemap-index recursion), common
sitemap locations, and RSS/Atom feeds; classifies each URL by content type; clusters by
URL slug; computes publishing cadence; and optionally deep-reads the top N pages for stats.
When to use
- "Catalog all the content on [domain]."
- Before an SEO audit, content-gap analysis, or as
content-brief-factory's existing-
coverage check.
- When you need funnel-stage mapping or publishing-pattern stats for a competitor/own site.
How to run
Bundled script scripts/site_catalog.py (Python 3 stdlib only — no install):
python3 ${SKILL_DIR}/scripts/site_catalog.py --domain example.com --output ${WORKSPACE}/catalog.json
| Flag |
Default |
Meaning |
--domain |
(required) |
example.com or https://example.com. |
--deep |
0 |
Deep-read the first N catalog URLs for page stats. |
--include-non-blog |
true |
false keeps only blog-posts. |
--output |
- |
JSON path (stdout if -). |
Examples
# Full inventory
python3 ${SKILL_DIR}/scripts/site_catalog.py --domain example.com --output ${WORKSPACE}/catalog.json
# Inventory + deep stats on the first 10 pages
python3 ${SKILL_DIR}/scripts/site_catalog.py --domain example.com --deep 10 --output ${WORKSPACE}/catalog.json
Outputs
JSON { summary, pages }:
summary: {domain, total_pages, sitemap_found, by_type, by_topic, publishing_cadence}
(sitemap_found: false is itself a negative SEO signal).pages[]: {url, title, date, type, topic_cluster, deep_analysis?}. deep_analysis
(when --deep) carries {word_count, image_count, internal_link_count, has_cta}.
The agent does the higher-order reasoning the contract calls for: LLM topic clustering
on large sites, classifying ambiguous URLs, and inferring each deep page's target keyword
and funnel stage (TOFU/MOFU/BOFU) from the stats. Produce the markdown summary table from
the JSON for the user.
Credentials / env
- Required: none — sitemap/robots/feeds/HTTP are all keyless.
- Optional:
ANTHROPIC_API_KEY / OPENAI_API_KEY — not used by the script (the agent is the model);
listed only because the contract's LLM clustering/funnel-inference is the agent's job.APIFY_API_TOKEN (paid, with a fallback) — If set → the agent orchestrates an Apify
extractor for JS-heavy / sitemap-missing sites (better coverage when keyless discovery is
blocked). If not set → fall back to the keyless sitemap/robots/feeds/HTTP crawl (the
default). Route large crawls through Robomotion Proxy at the node.
Notes & edge cases
- Sitemap.xml is the best source; a missing one is a (negative) SEO signal — reported as
sitemap_found: false. RSS only surfaces recent posts, so feeds supplement, not replace.
- Dedup is by canonical URL (tracking params + fragment stripped).
- Deep analysis is capped to
--deep to control time; the script throttles between pages.
- Type classification is rule-based on URL/title patterns; the agent should re-tag the
handful of ambiguous URLs.
1---2name: site-content-catalog3description: Crawl a site's sitemap, sitemap index, and RSS/Atom feeds to build a complete content inventory — every page with URL, title, date, content type, and topic cluster — plus publishing-cadence stats and optional deep per-page analysis (word count, images, internal links, has-CTA). No API key. Foundation for SEO audits, content-gap analysis, and brand-voice extraction.4---56# Site Content Catalog78Build a content inventory for any domain — keyless. The script discovers pages via9`robots.txt` Sitemap directives, `sitemap.xml` (with sitemap-index recursion), common10sitemap locations, and RSS/Atom feeds; classifies each URL by content type; clusters by11URL slug; computes publishing cadence; and optionally deep-reads the top N pages for stats.1213## When to use1415- "Catalog all the content on [domain]."16- Before an SEO audit, content-gap analysis, or as `content-brief-factory`'s existing-17 coverage check.18- When you need funnel-stage mapping or publishing-pattern stats for a competitor/own site.1920## How to run2122Bundled script `scripts/site_catalog.py` (Python 3 stdlib only — no install):2324```bash25python3 ${SKILL_DIR}/scripts/site_catalog.py --domain example.com --output ${WORKSPACE}/catalog.json26```2728| Flag | Default | Meaning |29|---|---|---|30| `--domain` | (required) | `example.com` or `https://example.com`. |31| `--deep` | `0` | Deep-read the first N catalog URLs for page stats. |32| `--include-non-blog` | `true` | `false` keeps only blog-posts. |33| `--output` | `-` | JSON path (stdout if `-`). |3435### Examples3637```bash38# Full inventory39python3 ${SKILL_DIR}/scripts/site_catalog.py --domain example.com --output ${WORKSPACE}/catalog.json4041# Inventory + deep stats on the first 10 pages42python3 ${SKILL_DIR}/scripts/site_catalog.py --domain example.com --deep 10 --output ${WORKSPACE}/catalog.json43```4445## Outputs4647JSON `{ summary, pages }`:4849- `summary`: `{domain, total_pages, sitemap_found, by_type, by_topic, publishing_cadence}`50 (`sitemap_found: false` is itself a negative SEO signal).51- `pages[]`: `{url, title, date, type, topic_cluster, deep_analysis?}`. `deep_analysis`52 (when `--deep`) carries `{word_count, image_count, internal_link_count, has_cta}`.5354The **agent** does the higher-order reasoning the contract calls for: LLM topic clustering55on large sites, classifying ambiguous URLs, and inferring each deep page's target keyword56and funnel stage (TOFU/MOFU/BOFU) from the stats. Produce the markdown summary table from57the JSON for the user.5859## Credentials / env6061- **Required:** none — sitemap/robots/feeds/HTTP are all keyless.62- **Optional:**63 - `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` — not used by the script (the agent is the model);64 listed only because the contract's LLM clustering/funnel-inference is the agent's job.65 - `APIFY_API_TOKEN` (paid, with a fallback) — If set → the agent orchestrates an Apify66 extractor for JS-heavy / sitemap-missing sites (better coverage when keyless discovery is67 blocked). If not set → fall back to the keyless sitemap/robots/feeds/HTTP crawl (the68 default). Route large crawls through Robomotion Proxy at the node.6970## Notes & edge cases7172- Sitemap.xml is the best source; a missing one is a (negative) SEO signal — reported as73 `sitemap_found: false`. RSS only surfaces recent posts, so feeds supplement, not replace.74- Dedup is by canonical URL (tracking params + fragment stripped).75- Deep analysis is capped to `--deep` to control time; the script throttles between pages.76- Type classification is rule-based on URL/title patterns; the agent should re-tag the77 handful of ambiguous URLs.
robomotionio