Keyword Research Pipeline (Master Orchestrator)
Provider ruling (2026-06-12 — supersedes the 2026-06-10 DataForSEO note)
Semrush MCP is the data layer. The paid plan has full MCP data access (verified live).
DataForSEO is retired; scripts/dataforseo_keyword_pipeline.py is archived and must not run.
Every layer's real data calls are pinned in
references/semrush-data-layer.md — read it before
dispatching any layer. The server has 13 tools and exactly one call pattern
(get_report_schema → execute_report); any mcp__semrush__<kebab-name> mention in the
layer briefs below is a historical fiction that the data-layer file overrides. Metric
thresholds still go through references/semrush-metric-translation.md.
Layer 1c (keyword-aio-gap) remains SKIPPED — this MCP server exposes no AI-visibility
toolkit. Do not fake aio_gap rows or aio_sov_competitor_top. Layer 3's AIO check is
presence-only via the Fk SERP-features column (see the data-layer file). Keep the
deterministic quality, judgment rewrite, and voice-drift rollback gates.
Take a brand and produce a vetted keyword queue (keyword-queue.csv) ready for the autonomous blog loop. Layered chain (0 → 1a → 1b → 1c → 1d → 2 → 3 → 4 → 5), each rejecting candidates with reasons logged, each dispatched as a fresh Agent.
This is the upstream of auto-blog-loop. The blog loop reads the queue this orchestrator emits — never the raw keyword-ideas.csv.
Threshold reminder. Every layer in this chain is recalibrated against .claude/skills/keyword-research-pipeline/references/dataforseo-metric-translation.md. If a brief tells a layer to apply a numeric threshold, the underlying skill's gate should already match the metric-translation doc — do not let an Agent invent its own threshold from training data. mcp__semrush__* and mcp__ahrefs__* are retired for keyword research; if any sub-agent reaches for either provider, that's a migration leftover bug, not a fallback.
Invocation
/keyword-research-pipeline [--regen] [--max-redteam 30]
--regen forces fresh runs of idempotent layers (topic-discovery re-runs even if brand-config hash unchanged, seed-modifier-prompt re-runs, redteam re-evaluates already-judged rows, BID/AIO refetch cached SERP data).
--max-redteam overrides the Layer 4 batch size (default 30).
Why agent dispatch, not skill fork
Same constraint as /blog-pipeline (line 22-24 of blog-pipeline/SKILL.md): the Skill tool forks with the parent's context. After any compaction or accumulated history, that fork hits Prompt is too long. The Agent tool starts each layer with a clean window. This is non-negotiable — every layer MUST be an Agent dispatch.
Process
Resolve project root (C:\Users\ndong\Downloads\blog-agent-2 or wherever the host runs). Set {ROOT}.
Layer 0 — /topic-discovery (Agent dispatch). Build a topic-graph + market-trends snapshot for the brand's category before any seed work happens. Self-contained brief:
You are running Layer 0 of the keyword research pipeline at {ROOT}.
Your job: produce content-pipeline/0-keywords/topic-graph.json plus content-pipeline/0-keywords/trends.md per .claude/skills/topic-discovery/SKILL.md. Read the SKILL.md first.
Read brand-config.md. Pull mcp__semrush__topic-research for the brand's category-level seeds and mcp__semrush__trends-overview for the brand domain (or category URL). Synthesize the top 5 KSB clusters per category and the trending vs. declining queries. Save the JSON + markdown atomically.
Idempotent: if topic-graph.json exists and the brand-config hash is unchanged AND --regen was not passed, exit without re-generating. Never block the pipeline — if Topic Research / .Trends both error, log to cache/topic-discovery-failed.log and exit cleanly with topic-graph.json containing only the brand-config seeds.
Return: top 5 cluster names, count of trending-up / trending-down queries, idempotency status (regenerated vs. skipped). Under 250 words.
On failure: log and continue. Layer 0 is enrichment, not a gate — Layer 1a will still run with brand-config seeds alone if topic-graph.json is empty.
Layer 1a — /seed-modifier-prompt (Agent dispatch). Self-contained brief:
You are running Layer 1a of the keyword research pipeline at {ROOT}.
Your job: produce content-pipeline/0-keywords/seeds.json per .claude/skills/seed-modifier-prompt/SKILL.md. Read the SKILL.md first.
Read brand-config.md AND content-pipeline/0-keywords/topic-graph.json (from Layer 0). Pre-feed the top 5 KSB clusters from topic-graph.json into the seed-generation prompt so seeds anchor in real Semrush clusters, not just brand-config alone. Generate 10 seeds + 10+ modifiers (with at least 3 AI-resistant: calculator/checker/generator/tool/template/examples). No word overlap between seeds and modifiers.
Idempotent: if seeds.json exists and brand-config hash unchanged, exit without re-generating. Force re-gen only if --regen passed.
Return: seeds count, modifiers count, tool-modifier count, first 3 seeds, whether re-generated or skipped. Under 200 words.
On failure: stop the pipeline. Layer 1a is the cheapest layer; if it fails, something is structurally wrong.
Layer 1b — /content-gap-analysis (Agent dispatch). Brief:
You are running Layer 1b at {ROOT}.
Your job: produce content-pipeline/0-keywords/keyword-ideas.csv per .claude/skills/content-gap-analysis/SKILL.md. Read the SKILL.md first.
Auto-discover competitors via mcp__semrush__organic-competitors if brand-config doesn't list any; cache to cache/competitors.json so Layer 1c reuses the same set. Read seeds.json from Layer 1a — for each seed, expand via mcp__semrush__keyword-magic-broad (and -phrase / -related where breadth is thin) with modifiers as `include` filter. Pull intents array via mcp__semrush__keyword-overview for each row.
Run multi-mode Keyword Gap via mcp__semrush__keyword-gap — all five modes (missing / weak / unique / common / strong). Tag every row with `gap_mode`. Route `strong`-mode rows to cache/strong-positions.csv (NOT the writing pool). Merge competitor-gap + seed-modifier sources, dedupe on keyword. Add `source` and `gap_mode` columns.
Apply Semrush KD% threshold ≤ 70 (default) — DO NOT transplant Ahrefs KD ≤ 60. The metric-translation doc has the recalibration math.
Return: total candidates, breakdown by source (competitor_gap / seed_modifier / both), breakdown by gap_mode (missing / weak / unique / common), strong-positions count, top 5 candidates by traffic_potential. Under 300 words.
On failure: stop. Layer 1b failures usually mean a Semrush auth issue (401/OAuth) or quota exhaustion (429).
Layer 1c — /keyword-aio-gap (Agent dispatch). Brief:
You are running Layer 1c at {ROOT}.
Your job: append AI-search gap candidates to content-pipeline/0-keywords/keyword-ideas.csv per .claude/skills/keyword-aio-gap/SKILL.md. Read the SKILL.md first.
Use the Semrush AI Toolkit's prompt-centric model (NOT the legacy Ahrefs Brand Radar keyword-centric model). For each brand + competitor, register the relevant prompts via mcp__semrush__ai-toolkit-prompts and call mcp__semrush__ai-toolkit-mentions across the multi-engine panel (AIO, ChatGPT, Gemini, Perplexity, Copilot). Compute the gap (competitor-cited prompts where brand isn't). Enrich with mcp__semrush__keyword-overview metrics. Append rows with source=aio_gap (or merge to existing rows as source=both). Add aio_engines and aio_sov_competitor_top columns for Layer 5 prioritization.
If AI Toolkit returns 429, exit code 75 — orchestrator retries on next cron cycle. Don't crash.
Return: brand mentions, total competitor mentions across the engine panel, gap-prompt count, top 5 gap keywords with their dominant engine. Under 300 words.
On Layer 1c rate-limit: don't fail the pipeline — Layer 1b already produced a candidate pool, downstream layers can vet it. Note the 1c skip in the run summary.
Layer 1d — /keyword-question-mining (Agent dispatch). Brief:
You are running Layer 1d at {ROOT}.
Your job: append question-shaped keyword candidates to content-pipeline/0-keywords/keyword-ideas.csv per .claude/skills/keyword-question-mining/SKILL.md. Read the SKILL.md first.
For every primary seed in seeds.json, call mcp__semrush__keyword-magic-questions to pull the question-form variants Semrush exposes. Then call mcp__semrush__serp-results per top seed and parse the People Also Ask block for additional questions Keyword Magic missed. Cap the merged pool at 100 rows per pipeline run; dedupe against existing keyword-ideas.csv. Tag every appended row with source=question_mining and a question_subtype field (`paa` / `km_question` / `both`). Pull intents via mcp__semrush__keyword-overview for each new row.
On 429: persist progress, exit 75. The orchestrator handles retry.
Return: km_questions count, paa-only count, deduped-against-existing count, top 5 questions by volume, subtype breakdown. Under 250 words.
On failure (other than 429): continue. Layer 1d is enrichment; missing question-mining rows don't break the queue.
Layer 2 — /keyword-vet-bid (Agent dispatch). Brief:
You are running Layer 2 at {ROOT}.
Your job: enrich content-pipeline/0-keywords/keyword-ideas.csv with BID verdicts per .claude/skills/keyword-vet-bid/SKILL.md. Read the SKILL.md first.
Resolve brand AS via mcp__semrush__domain-overview, cache to cache/brand-as.json (7-day TTL). Delete any leftover cache/brand-dr.json — its DR-shaped payload is incompatible.
For each row: compute brand_fit + product_fit; classify intent — PRIMARY signal is the Semrush `intents` array from mcp__semrush__keyword-overview (informational/commercial = PASS; transactional/navigational = FAIL); fall back to URL-pattern heuristic on mcp__semrush__serp-results only when `intents` is empty/mixed. Pull per-URL Authority Score for the SERP top-10 via mcp__semrush__serp-overview / mcp__semrush__domain-overview. Apply the BID gate (recalibrated for Semrush — as_top10_median ≤ brand_AS + 12; weak_link_count of pages with AS < brand_AS + 4; KD% ≤ 70 baseline).
Persist columns: brand_fit, product_fit, serp_intent, as_top10_median, weak_link_count, bid_verdict, bid_reason.
Calibration check: if all rows pass OR all fail, log to cache/bid-calibration.log and adjust thresholds per the SKILL's "Calibration" section.
Return: total vetted, B/I/D pass rates, share of intent decisions made via `intents` array vs URL fallback, overall PASS count, top 5 PASS by traffic_potential, top 5 FAIL with reasons. Under 350 words.
On failure: stop. BID failures usually mean Semrush quota or auth issues — not recoverable mid-run.
Layer 3 — /keyword-vet-aio (Agent dispatch). Brief:
You are running Layer 3 at {ROOT}.
Your job: enrich BID-PASS rows in keyword-ideas.csv with AIO cannibalization verdicts per .claude/skills/keyword-vet-aio/SKILL.md. Read the SKILL.md and the rubric at .claude/skills/keyword-research-pipeline/references/aio-cannibalization-rubric.md first.
For each BID-PASS row: detect AIO presence via mcp__semrush__serp-overview — Semrush's literal SERP-feature key is `ai_overview` (underscore — NOT the legacy Ahrefs `ai-overview`). Apply exemptions (tool-led, commercial-investigation, aio_gap source). For non-exempt AIO-present rows fetch the AIO body in this order: mcp__semrush__ai-toolkit-response → mcp__semrush__serp-overview's serp-features summary → WebFetch on https://www.google.com/search?q=… Spawn the adversarial Sonnet sub-agent with the 0-10 completeness brief; persist has_aio, aio_completeness_score, aio_click_intent, aio_verdict, aio_reasoning, aio_body_source.
Cache AIO bodies under cache/aio-fetch/ with _meta.source. Refresh weekly. Pre-migration cache files (brand_radar_*) are stale — re-fetch.
Calibration check: if every score is 8+ OR every score is 4-, the scorer is mis-calibrated — log and re-run with strengthened brief per the SKILL's "Calibration" section.
Return: total checked, breakdown (no AIO / exempt / PASS / RISKY / FAIL_CANNIBALIZED / UNKNOWN), body-source mix (ai_toolkit / serp_features / webfetch), top 3 cannibalized rejections, top 3 risky survivors. Under 400 words.
On rate-limit (exit 75): persist progress, surface to orchestrator. Layer 3 is the most quota-heavy layer (one serp-overview + one ai-toolkit-response per non-exempt BID-PASS row).
Layer 4 — /keyword-redteam (Agent dispatch — judgment-heavy, stays on parent model). Brief:
You are running Layer 4 at {ROOT}.
Your job: redteam top-{max_redteam} survivors per .claude/skills/keyword-redteam/SKILL.md. Read the SKILL.md first.
Filter to bid_verdict=PASS AND aio_verdict ∈ {PASS, RISKY, UNKNOWN} AND no existing redteam_verdict (idempotent). Take top 30 by current priority_score. Spawn the skeptical-SEO sub-agent with the (a)-(d) challenge brief. The agent reasons over Semrush metrics (intents array, AS gap, KD%, KSB cluster id, AI Toolkit SoV) — not Ahrefs DR/KD heuristics.
Apply discipline check: if KEEP rate > 80% with weak per-candidate critique (avg < 60 words), re-run with the contrarian addendum from the SKILL.
Persist redteam_verdict, redteam_priority_delta, redteam_critique_summary to keyword-ideas.csv. Save full critique to redteam-notes.md.
Return: KEEP/DROP/REVISE counts, sum of priority deltas, top 3 DROPs with one-line reasons, top 3 REVISE-down with reasons. Under 400 words.
On failure: stop. Don't ship a queue without the redteam check.
Layer 5 — /keyword-prioritization (Agent dispatch). Brief:
You are running Layer 5 at {ROOT}.
Your job: emit content-pipeline/0-keywords/keyword-queue.csv per the updated .claude/skills/keyword-prioritization/SKILL.md. Read the SKILL.md first.
Filter pool: only rows where bid_verdict=PASS AND aio_verdict ∈ {PASS, RISKY} AND redteam_verdict ∈ {KEEP, REVISE_PRIORITY}.
Apply the existing 0.4/0.3/0.3 scoring weights. Apply boosts:
+1.5 if source=aio_gap
+0.5 if gap_mode=weak
+0.7 if gap_mode=unique
-0.3 if gap_mode=common
+0.5 if cluster_authority_gap=true (KSB cluster the brand has zero authority in but is low-difficulty)
+1.0 if serp_intent=tool-led (but ROUTE those to tool-opportunities.csv, not the writing queue)
Tie-breaker on equal priority_score: row with higher aio_sov_competitor_top wins.
Apply redteam_priority_delta (capped ±2.0).
Re-rank, write top-50 to keyword-queue.csv. Tool-opportunity rows go to tool-opportunities.csv (not the writing queue). gap_mode=strong rows are already in cache/strong-positions.csv and are ignored here.
Return: queue size, top 10 with rank/keyword/priority_score/source/intent/gap_mode/verdicts, tool-opportunities count. Under 350 words.
Verify each layer's output file exists before advancing. Layer 0: topic-graph.json + trends.md (or topic-discovery-failed.log). Layer 1a: seeds.json. Layer 1b/1c/1d: keyword-ideas.csv with relevant source rows; cache/strong-positions.csv when applicable. Layer 2/3: required columns present. Layer 4: redteam-notes.md exists, keyword-ideas.csv has redteam columns. Layer 5: keyword-queue.csv exists.
Reporting. When complete, output:
✓ Keyword research pipeline complete
Layers:
✓ 0 topic-discovery → topic-graph.json (12 clusters, 8 trending-up) / trends.md
✓ 1a seed-modifier → seeds.json (10 seeds, 12 modifiers, 4 tool-modifiers)
✓ 1b content-gap → keyword-ideas.csv (124 candidates: 78 missing, 22 weak, 14 unique, 10 common; 6 strong → cache)
✓ 1c keyword-aio-gap → +37 rows from aio_gap (engines: AIO=18 ChatGPT=11 Perplexity=8) (or SKIPPED: rate-limit)
✓ 1d question-mining → +42 rows (28 km_questions, 19 paa, dedup-overlap=5)
✓ 2 keyword-vet-bid → 203 vetted, 91 PASS / 112 FAIL (intents-array signal: 78%, URL-fallback: 22%)
✓ 3 keyword-vet-aio → 91 checked, 71 PASS / 14 RISKY / 4 FAIL_CANNIBALIZED (body source: ai_toolkit=58 serp=7 webfetch=2)
✓ 4 keyword-redteam → 30 redteamed, 22 KEEP / 5 DROP / 3 REVISE
✓ 5 prioritization → keyword-queue.csv (top 50 ranked, 8 aio_gap-boosted, 4 cluster-authority-boosted)
→ tool-opportunities.csv (12 entries)
Queue ready for /auto-blog-loop.
Failure handling
Per the plan, bad keyword research is worse than empty queue. Layer failures stop the chain rather than auto-retrying. Specifically:
- Layer 0 fails → continue. Topic-discovery is enrichment, not a gate.
- Layer 1a fails → stop. Brand-config hash issue or agent malfunction.
- Layer 1b fails → stop. Likely Semrush auth issue (401) or unauthorized OAuth.
- Layer 1c fails (rate-limit) → continue without 1c rows; log skip.
- Layer 1c fails (other) → continue with a warning; 1b alone is a viable candidate pool.
- Layer 1d fails (rate-limit, exit 75) → persist progress; orchestrator retries on next cron.
- Layer 1d fails (other) → continue. Question mining is enrichment.
- Layer 2 fails → stop. The mechanical filter is required.
- Layer 3 rate-limit (exit 75) → persist progress, exit cleanly with code 75. The auto-blog-loop reads this and retries on the next cron cycle.
- Layer 3 fails (other) → stop. AIO check is required.
- Layer 4 fails → stop. Redteam is required.
- Layer 5 fails → stop. Queue emission is the whole point.
The orchestrator never auto-retries — bad keyword research compounds. It logs the failure to content-pipeline/0-keywords/cache/pipeline-failures.log and exits.
When /auto-blog-loop invokes this orchestrator
The blog loop calls this when keyword-queue.csv is empty (selector exit 2). The keyword-research-pipeline runs to completion (or fails cleanly), then control returns to the blog loop. If it fails:
- The previous
keyword-queue.csv (if any) stays in place — blog loop falls back to the stale queue.
- If no previous queue, blog loop exits cleanly with a "no work" log entry.
Calibration cadence
The mechanical layers (BID, AIO) need calibration on first run after Semrush migration AND after major brand-config changes. Calibration logs are in cache/:
cache/bid-calibration.log — threshold drift, pass-rate trendscache/aio-calibration.log — score distribution, rubber-stamp detection
The runbook (.claude/skills/auto-blog-loop/references/runbook.md) has a section on reading these logs and tuning thresholds. Always recalibrate against references/semrush-metric-translation.md — Ahrefs thresholds will silently mis-fire on Semrush data.
Cost per run
Approximate, for a 200-candidate pool, on Semrush quota:
- Layer 0: ~3-5 Semrush MCP calls (Topic Research + .Trends overview); idempotent — most runs skip
- Layer 1a: ~1K LLM tokens (one Agent call)
- Layer 1b: ~10-15 Semrush MCP calls (organic-competitors + 5 keyword-gap modes + ~6 keyword-magic expansions per seed)
- Layer 1c: ~10-30 AI Toolkit calls (mentions across the engine panel; weighted heavier than the legacy Brand Radar count because of multi-engine fan-out)
- Layer 1d: ~10-20 keyword-magic-questions + serp-results PAA-extraction calls
- Layer 2: ~200 keyword-overview + 200 serp-overview + ~200-2000 domain-overview calls (batched for the per-domain AS lookups)
- Layer 3: ~100 serp-overview + ~50-70 ai-toolkit-response + ~50 Sonnet sub-agent invocations
- Layer 4: ~1 Opus call (30K-50K tokens)
- Layer 5: ~1 Agent call
Total: ~$0.50-1.50 in Semrush MCP quota + ~$0.50 in LLM tokens per run. Weekly cadence keeps annual cost < $80. The blog loop's article cost is much higher; keyword research is cheap insurance. Layer 0 + Layer 1d add ~$0.05-0.10 per run on top of the legacy chain — small price for the upstream uplift.
1---2name: keyword-research-pipeline3description: Master orchestrator for the keyword research pipeline. Chains topic-discovery → seed/modifier ideation → competitor + AI-search gap analysis → question mining → BID method → AIO cannibalization check → adversarial redteam → final ranked queue. Same anti-context-bloat pattern as /blog-pipeline (every layer is an Agent dispatch, never a Skill fork).4---56# Keyword Research Pipeline (Master Orchestrator)78## Provider ruling (2026-06-12 — supersedes the 2026-06-10 DataForSEO note)910**Semrush MCP is the data layer.** The paid plan has full MCP data access (verified live).11DataForSEO is retired; `scripts/dataforseo_keyword_pipeline.py` is archived and must not run.1213Every layer's real data calls are pinned in14[`references/semrush-data-layer.md`](./references/semrush-data-layer.md) — read it before15dispatching any layer. The server has 13 tools and exactly one call pattern16(`get_report_schema` → `execute_report`); any `mcp__semrush__<kebab-name>` mention in the17layer briefs below is a historical fiction that the data-layer file overrides. Metric18thresholds still go through `references/semrush-metric-translation.md`.1920Layer 1c (`keyword-aio-gap`) remains **SKIPPED** — this MCP server exposes no AI-visibility21toolkit. Do not fake `aio_gap` rows or `aio_sov_competitor_top`. Layer 3's AIO check is22presence-only via the `Fk` SERP-features column (see the data-layer file). Keep the23deterministic quality, judgment rewrite, and voice-drift rollback gates.2425Take a brand and produce a *vetted* keyword queue (`keyword-queue.csv`) ready for the autonomous blog loop. Layered chain (0 → 1a → 1b → 1c → 1d → 2 → 3 → 4 → 5), each rejecting candidates with reasons logged, each dispatched as a fresh Agent.2627This is the upstream of `auto-blog-loop`. The blog loop reads the queue this orchestrator emits — never the raw `keyword-ideas.csv`.2829> **Threshold reminder.** Every layer in this chain is recalibrated against `.claude/skills/keyword-research-pipeline/references/dataforseo-metric-translation.md`. If a brief tells a layer to apply a numeric threshold, the underlying skill's gate should already match the metric-translation doc — do not let an Agent invent its own threshold from training data. **`mcp__semrush__*` and `mcp__ahrefs__*` are retired for keyword research**; if any sub-agent reaches for either provider, that's a migration leftover bug, not a fallback.3031## Invocation3233```34/keyword-research-pipeline [--regen] [--max-redteam 30]35```3637`--regen` forces fresh runs of idempotent layers (topic-discovery re-runs even if brand-config hash unchanged, seed-modifier-prompt re-runs, redteam re-evaluates already-judged rows, BID/AIO refetch cached SERP data).3839`--max-redteam` overrides the Layer 4 batch size (default 30).4041## Why agent dispatch, not skill fork4243Same constraint as `/blog-pipeline` (line 22-24 of `blog-pipeline/SKILL.md`): the Skill tool forks with the parent's context. After any compaction or accumulated history, that fork hits `Prompt is too long`. The Agent tool starts each layer with a clean window. This is non-negotiable — every layer MUST be an Agent dispatch.4445## Process46471. **Resolve project root** (`C:\Users\ndong\Downloads\blog-agent-2` or wherever the host runs). Set `{ROOT}`.48492. **Layer 0 — `/topic-discovery`** (Agent dispatch). Build a topic-graph + market-trends snapshot for the brand's category before any seed work happens. Self-contained brief:5051 ```52 You are running Layer 0 of the keyword research pipeline at {ROOT}.5354 Your job: produce content-pipeline/0-keywords/topic-graph.json plus content-pipeline/0-keywords/trends.md per .claude/skills/topic-discovery/SKILL.md. Read the SKILL.md first.5556 Read brand-config.md. Pull mcp__semrush__topic-research for the brand's category-level seeds and mcp__semrush__trends-overview for the brand domain (or category URL). Synthesize the top 5 KSB clusters per category and the trending vs. declining queries. Save the JSON + markdown atomically.5758 Idempotent: if topic-graph.json exists and the brand-config hash is unchanged AND --regen was not passed, exit without re-generating. Never block the pipeline — if Topic Research / .Trends both error, log to cache/topic-discovery-failed.log and exit cleanly with topic-graph.json containing only the brand-config seeds.5960 Return: top 5 cluster names, count of trending-up / trending-down queries, idempotency status (regenerated vs. skipped). Under 250 words.61 ```6263 On failure: log and continue. Layer 0 is enrichment, not a gate — Layer 1a will still run with brand-config seeds alone if topic-graph.json is empty.64653. **Layer 1a — `/seed-modifier-prompt`** (Agent dispatch). Self-contained brief:6667 ```68 You are running Layer 1a of the keyword research pipeline at {ROOT}.6970 Your job: produce content-pipeline/0-keywords/seeds.json per .claude/skills/seed-modifier-prompt/SKILL.md. Read the SKILL.md first.7172 Read brand-config.md AND content-pipeline/0-keywords/topic-graph.json (from Layer 0). Pre-feed the top 5 KSB clusters from topic-graph.json into the seed-generation prompt so seeds anchor in real Semrush clusters, not just brand-config alone. Generate 10 seeds + 10+ modifiers (with at least 3 AI-resistant: calculator/checker/generator/tool/template/examples). No word overlap between seeds and modifiers.7374 Idempotent: if seeds.json exists and brand-config hash unchanged, exit without re-generating. Force re-gen only if --regen passed.7576 Return: seeds count, modifiers count, tool-modifier count, first 3 seeds, whether re-generated or skipped. Under 200 words.77 ```7879 On failure: stop the pipeline. Layer 1a is the cheapest layer; if it fails, something is structurally wrong.80814. **Layer 1b — `/content-gap-analysis`** (Agent dispatch). Brief:8283 ```84 You are running Layer 1b at {ROOT}.8586 Your job: produce content-pipeline/0-keywords/keyword-ideas.csv per .claude/skills/content-gap-analysis/SKILL.md. Read the SKILL.md first.8788 Auto-discover competitors via mcp__semrush__organic-competitors if brand-config doesn't list any; cache to cache/competitors.json so Layer 1c reuses the same set. Read seeds.json from Layer 1a — for each seed, expand via mcp__semrush__keyword-magic-broad (and -phrase / -related where breadth is thin) with modifiers as `include` filter. Pull intents array via mcp__semrush__keyword-overview for each row.8990 Run multi-mode Keyword Gap via mcp__semrush__keyword-gap — all five modes (missing / weak / unique / common / strong). Tag every row with `gap_mode`. Route `strong`-mode rows to cache/strong-positions.csv (NOT the writing pool). Merge competitor-gap + seed-modifier sources, dedupe on keyword. Add `source` and `gap_mode` columns.9192 Apply Semrush KD% threshold ≤ 70 (default) — DO NOT transplant Ahrefs KD ≤ 60. The metric-translation doc has the recalibration math.9394 Return: total candidates, breakdown by source (competitor_gap / seed_modifier / both), breakdown by gap_mode (missing / weak / unique / common), strong-positions count, top 5 candidates by traffic_potential. Under 300 words.95 ```9697 On failure: stop. Layer 1b failures usually mean a Semrush auth issue (401/OAuth) or quota exhaustion (429).98995. **Layer 1c — `/keyword-aio-gap`** (Agent dispatch). Brief:100101 ```102 You are running Layer 1c at {ROOT}.103104 Your job: append AI-search gap candidates to content-pipeline/0-keywords/keyword-ideas.csv per .claude/skills/keyword-aio-gap/SKILL.md. Read the SKILL.md first.105106 Use the Semrush AI Toolkit's prompt-centric model (NOT the legacy Ahrefs Brand Radar keyword-centric model). For each brand + competitor, register the relevant prompts via mcp__semrush__ai-toolkit-prompts and call mcp__semrush__ai-toolkit-mentions across the multi-engine panel (AIO, ChatGPT, Gemini, Perplexity, Copilot). Compute the gap (competitor-cited prompts where brand isn't). Enrich with mcp__semrush__keyword-overview metrics. Append rows with source=aio_gap (or merge to existing rows as source=both). Add aio_engines and aio_sov_competitor_top columns for Layer 5 prioritization.107108 If AI Toolkit returns 429, exit code 75 — orchestrator retries on next cron cycle. Don't crash.109110 Return: brand mentions, total competitor mentions across the engine panel, gap-prompt count, top 5 gap keywords with their dominant engine. Under 300 words.111 ```112113 On Layer 1c rate-limit: don't fail the pipeline — Layer 1b already produced a candidate pool, downstream layers can vet it. Note the 1c skip in the run summary.1141156. **Layer 1d — `/keyword-question-mining`** (Agent dispatch). Brief:116117 ```118 You are running Layer 1d at {ROOT}.119120 Your job: append question-shaped keyword candidates to content-pipeline/0-keywords/keyword-ideas.csv per .claude/skills/keyword-question-mining/SKILL.md. Read the SKILL.md first.121122 For every primary seed in seeds.json, call mcp__semrush__keyword-magic-questions to pull the question-form variants Semrush exposes. Then call mcp__semrush__serp-results per top seed and parse the People Also Ask block for additional questions Keyword Magic missed. Cap the merged pool at 100 rows per pipeline run; dedupe against existing keyword-ideas.csv. Tag every appended row with source=question_mining and a question_subtype field (`paa` / `km_question` / `both`). Pull intents via mcp__semrush__keyword-overview for each new row.123124 On 429: persist progress, exit 75. The orchestrator handles retry.125126 Return: km_questions count, paa-only count, deduped-against-existing count, top 5 questions by volume, subtype breakdown. Under 250 words.127 ```128129 On failure (other than 429): continue. Layer 1d is enrichment; missing question-mining rows don't break the queue.1301317. **Layer 2 — `/keyword-vet-bid`** (Agent dispatch). Brief:132133 ```134 You are running Layer 2 at {ROOT}.135136 Your job: enrich content-pipeline/0-keywords/keyword-ideas.csv with BID verdicts per .claude/skills/keyword-vet-bid/SKILL.md. Read the SKILL.md first.137138 Resolve brand AS via mcp__semrush__domain-overview, cache to cache/brand-as.json (7-day TTL). Delete any leftover cache/brand-dr.json — its DR-shaped payload is incompatible.139140 For each row: compute brand_fit + product_fit; classify intent — PRIMARY signal is the Semrush `intents` array from mcp__semrush__keyword-overview (informational/commercial = PASS; transactional/navigational = FAIL); fall back to URL-pattern heuristic on mcp__semrush__serp-results only when `intents` is empty/mixed. Pull per-URL Authority Score for the SERP top-10 via mcp__semrush__serp-overview / mcp__semrush__domain-overview. Apply the BID gate (recalibrated for Semrush — as_top10_median ≤ brand_AS + 12; weak_link_count of pages with AS < brand_AS + 4; KD% ≤ 70 baseline).141142 Persist columns: brand_fit, product_fit, serp_intent, as_top10_median, weak_link_count, bid_verdict, bid_reason.143144 Calibration check: if all rows pass OR all fail, log to cache/bid-calibration.log and adjust thresholds per the SKILL's "Calibration" section.145146 Return: total vetted, B/I/D pass rates, share of intent decisions made via `intents` array vs URL fallback, overall PASS count, top 5 PASS by traffic_potential, top 5 FAIL with reasons. Under 350 words.147 ```148149 On failure: stop. BID failures usually mean Semrush quota or auth issues — not recoverable mid-run.1501518. **Layer 3 — `/keyword-vet-aio`** (Agent dispatch). Brief:152153 ```154 You are running Layer 3 at {ROOT}.155156 Your job: enrich BID-PASS rows in keyword-ideas.csv with AIO cannibalization verdicts per .claude/skills/keyword-vet-aio/SKILL.md. Read the SKILL.md and the rubric at .claude/skills/keyword-research-pipeline/references/aio-cannibalization-rubric.md first.157158 For each BID-PASS row: detect AIO presence via mcp__semrush__serp-overview — Semrush's literal SERP-feature key is `ai_overview` (underscore — NOT the legacy Ahrefs `ai-overview`). Apply exemptions (tool-led, commercial-investigation, aio_gap source). For non-exempt AIO-present rows fetch the AIO body in this order: mcp__semrush__ai-toolkit-response → mcp__semrush__serp-overview's serp-features summary → WebFetch on https://www.google.com/search?q=… Spawn the adversarial Sonnet sub-agent with the 0-10 completeness brief; persist has_aio, aio_completeness_score, aio_click_intent, aio_verdict, aio_reasoning, aio_body_source.159160 Cache AIO bodies under cache/aio-fetch/ with _meta.source. Refresh weekly. Pre-migration cache files (brand_radar_*) are stale — re-fetch.161162 Calibration check: if every score is 8+ OR every score is 4-, the scorer is mis-calibrated — log and re-run with strengthened brief per the SKILL's "Calibration" section.163164 Return: total checked, breakdown (no AIO / exempt / PASS / RISKY / FAIL_CANNIBALIZED / UNKNOWN), body-source mix (ai_toolkit / serp_features / webfetch), top 3 cannibalized rejections, top 3 risky survivors. Under 400 words.165 ```166167 On rate-limit (exit 75): persist progress, surface to orchestrator. Layer 3 is the most quota-heavy layer (one serp-overview + one ai-toolkit-response per non-exempt BID-PASS row).1681699. **Layer 4 — `/keyword-redteam`** (Agent dispatch — judgment-heavy, stays on parent model). Brief:170171 ```172 You are running Layer 4 at {ROOT}.173174 Your job: redteam top-{max_redteam} survivors per .claude/skills/keyword-redteam/SKILL.md. Read the SKILL.md first.175176 Filter to bid_verdict=PASS AND aio_verdict ∈ {PASS, RISKY, UNKNOWN} AND no existing redteam_verdict (idempotent). Take top 30 by current priority_score. Spawn the skeptical-SEO sub-agent with the (a)-(d) challenge brief. The agent reasons over Semrush metrics (intents array, AS gap, KD%, KSB cluster id, AI Toolkit SoV) — not Ahrefs DR/KD heuristics.177178 Apply discipline check: if KEEP rate > 80% with weak per-candidate critique (avg < 60 words), re-run with the contrarian addendum from the SKILL.179180 Persist redteam_verdict, redteam_priority_delta, redteam_critique_summary to keyword-ideas.csv. Save full critique to redteam-notes.md.181182 Return: KEEP/DROP/REVISE counts, sum of priority deltas, top 3 DROPs with one-line reasons, top 3 REVISE-down with reasons. Under 400 words.183 ```184185 On failure: stop. Don't ship a queue without the redteam check.18618710. **Layer 5 — `/keyword-prioritization`** (Agent dispatch). Brief:188189 ```190 You are running Layer 5 at {ROOT}.191192 Your job: emit content-pipeline/0-keywords/keyword-queue.csv per the updated .claude/skills/keyword-prioritization/SKILL.md. Read the SKILL.md first.193194 Filter pool: only rows where bid_verdict=PASS AND aio_verdict ∈ {PASS, RISKY} AND redteam_verdict ∈ {KEEP, REVISE_PRIORITY}.195196 Apply the existing 0.4/0.3/0.3 scoring weights. Apply boosts:197 +1.5 if source=aio_gap198 +0.5 if gap_mode=weak199 +0.7 if gap_mode=unique200 -0.3 if gap_mode=common201 +0.5 if cluster_authority_gap=true (KSB cluster the brand has zero authority in but is low-difficulty)202 +1.0 if serp_intent=tool-led (but ROUTE those to tool-opportunities.csv, not the writing queue)203 Tie-breaker on equal priority_score: row with higher aio_sov_competitor_top wins.204205 Apply redteam_priority_delta (capped ±2.0).206207 Re-rank, write top-50 to keyword-queue.csv. Tool-opportunity rows go to tool-opportunities.csv (not the writing queue). gap_mode=strong rows are already in cache/strong-positions.csv and are ignored here.208209 Return: queue size, top 10 with rank/keyword/priority_score/source/intent/gap_mode/verdicts, tool-opportunities count. Under 350 words.210 ```21121211. **Verify each layer's output file exists** before advancing. Layer 0: topic-graph.json + trends.md (or topic-discovery-failed.log). Layer 1a: seeds.json. Layer 1b/1c/1d: keyword-ideas.csv with relevant `source` rows; cache/strong-positions.csv when applicable. Layer 2/3: required columns present. Layer 4: redteam-notes.md exists, keyword-ideas.csv has redteam columns. Layer 5: keyword-queue.csv exists.21321412. **Reporting.** When complete, output:215216 ```217 ✓ Keyword research pipeline complete218219 Layers:220 ✓ 0 topic-discovery → topic-graph.json (12 clusters, 8 trending-up) / trends.md221 ✓ 1a seed-modifier → seeds.json (10 seeds, 12 modifiers, 4 tool-modifiers)222 ✓ 1b content-gap → keyword-ideas.csv (124 candidates: 78 missing, 22 weak, 14 unique, 10 common; 6 strong → cache)223 ✓ 1c keyword-aio-gap → +37 rows from aio_gap (engines: AIO=18 ChatGPT=11 Perplexity=8) (or SKIPPED: rate-limit)224 ✓ 1d question-mining → +42 rows (28 km_questions, 19 paa, dedup-overlap=5)225 ✓ 2 keyword-vet-bid → 203 vetted, 91 PASS / 112 FAIL (intents-array signal: 78%, URL-fallback: 22%)226 ✓ 3 keyword-vet-aio → 91 checked, 71 PASS / 14 RISKY / 4 FAIL_CANNIBALIZED (body source: ai_toolkit=58 serp=7 webfetch=2)227 ✓ 4 keyword-redteam → 30 redteamed, 22 KEEP / 5 DROP / 3 REVISE228 ✓ 5 prioritization → keyword-queue.csv (top 50 ranked, 8 aio_gap-boosted, 4 cluster-authority-boosted)229 → tool-opportunities.csv (12 entries)230231 Queue ready for /auto-blog-loop.232 ```233234## Failure handling235236Per the plan, **bad keyword research is worse than empty queue**. Layer failures stop the chain rather than auto-retrying. Specifically:237238- Layer 0 fails → continue. Topic-discovery is enrichment, not a gate.239- Layer 1a fails → stop. Brand-config hash issue or agent malfunction.240- Layer 1b fails → stop. Likely Semrush auth issue (401) or unauthorized OAuth.241- Layer 1c fails (rate-limit) → continue without 1c rows; log skip.242- Layer 1c fails (other) → continue with a warning; 1b alone is a viable candidate pool.243- Layer 1d fails (rate-limit, exit 75) → persist progress; orchestrator retries on next cron.244- Layer 1d fails (other) → continue. Question mining is enrichment.245- Layer 2 fails → stop. The mechanical filter is required.246- Layer 3 rate-limit (exit 75) → persist progress, exit cleanly with code 75. The auto-blog-loop reads this and retries on the next cron cycle.247- Layer 3 fails (other) → stop. AIO check is required.248- Layer 4 fails → stop. Redteam is required.249- Layer 5 fails → stop. Queue emission is the whole point.250251The orchestrator never auto-retries — bad keyword research compounds. It logs the failure to `content-pipeline/0-keywords/cache/pipeline-failures.log` and exits.252253## When `/auto-blog-loop` invokes this orchestrator254255The blog loop calls this when `keyword-queue.csv` is empty (selector exit 2). The keyword-research-pipeline runs to completion (or fails cleanly), then control returns to the blog loop. If it fails:256- The previous `keyword-queue.csv` (if any) stays in place — blog loop falls back to the stale queue.257- If no previous queue, blog loop exits cleanly with a "no work" log entry.258259## Calibration cadence260261The mechanical layers (BID, AIO) need calibration on first run after Semrush migration AND after major brand-config changes. Calibration logs are in `cache/`:262- `cache/bid-calibration.log` — threshold drift, pass-rate trends263- `cache/aio-calibration.log` — score distribution, rubber-stamp detection264265The runbook (`.claude/skills/auto-blog-loop/references/runbook.md`) has a section on reading these logs and tuning thresholds. **Always recalibrate against `references/semrush-metric-translation.md` — Ahrefs thresholds will silently mis-fire on Semrush data.**266267## Cost per run268269Approximate, for a 200-candidate pool, on Semrush quota:270271- Layer 0: ~3-5 Semrush MCP calls (Topic Research + .Trends overview); idempotent — most runs skip272- Layer 1a: ~1K LLM tokens (one Agent call)273- Layer 1b: ~10-15 Semrush MCP calls (organic-competitors + 5 keyword-gap modes + ~6 keyword-magic expansions per seed)274- Layer 1c: ~10-30 AI Toolkit calls (mentions across the engine panel; weighted heavier than the legacy Brand Radar count because of multi-engine fan-out)275- Layer 1d: ~10-20 keyword-magic-questions + serp-results PAA-extraction calls276- Layer 2: ~200 keyword-overview + 200 serp-overview + ~200-2000 domain-overview calls (batched for the per-domain AS lookups)277- Layer 3: ~100 serp-overview + ~50-70 ai-toolkit-response + ~50 Sonnet sub-agent invocations278- Layer 4: ~1 Opus call (30K-50K tokens)279- Layer 5: ~1 Agent call280281Total: ~$0.50-1.50 in Semrush MCP quota + ~$0.50 in LLM tokens per run. Weekly cadence keeps annual cost < $80. The blog loop's article cost is much higher; keyword research is cheap insurance. Layer 0 + Layer 1d add ~$0.05-0.10 per run on top of the legacy chain — small price for the upstream uplift.
lionelndong