Keyword Question Mining

Keyword Question Mining Skill (Layer 1d)

DATA LAYER RULING (2026-06-12). Every mcp__semrush__<kebab-name> tool named below is FICTIONAL (never existed on the live server), and DataForSEO is retired. Before making any data call, read ../keyword-research-pipeline/references/semrush-data-layer.md — it maps this layer's old calls to the real Semrush MCP pattern (get_report_schema -> execute_report). The logic below (filters, thresholds, output schema) remains binding; only the data calls changed.

Extract question-form keywords from the brand's category and the existing candidate pool. Question-form keywords have less competition, higher informational intent, and map cleanly to PAA-driven SERP real estate. Ryan Law explicitly flags the under-delivery of question keywords as a gap in the legacy method — Layer 1d closes it.

This is a first-class source, not a bolt-on. The rows it appends become regular candidates that flow through Layer 2 (BID), Layer 3 (AIO), Layer 4 (redteam), and Layer 5 (prioritization) like any other.

Input

/keyword-question-mining

Reads:

• content-pipeline/0-keywords/seeds.json — surviving seeds from Layer 1a
• content-pipeline/0-keywords/keyword-ideas.csv — current candidate pool (Layers 1b + 1c output)
• brand-config.md — used only to read the country code for SERP results (defaults to US)

No CLI flags. The skill is deterministic given its inputs and the global cap.

Process

1. Load inputs.

   • Parse seeds.json — extract the seeds array. If the file is missing, skip Source A (Keyword Magic Questions) and proceed with Source B alone, logging the absence.
   • Read keyword-ideas.csv. If the file doesn't exist, write a stub header (matching the contract in content-gap-analysis/SKILL.md step 9) plus the new question_subtype column, then proceed — Source B will run on an empty top-30 pool and produce zero rows; Source A still adds value.

2. Ensure the question_subtype column exists. If keyword-ideas.csv already has rows but no question_subtype header, add the column to the header row and backfill empty values for every existing row. Use atomic write (write to .tmp then move) to avoid corruption mid-edit.

3. Source A — Keyword Magic Questions per seed. For each seed in seeds.json:

   • Call mcp__semrush__keyword-magic-questions with the seed as the keyword. Request question-form variants with volume + KD% + intents array.
   • Drop rows with volume < 10 (the cheatsheet's filtering rule for Keyword Magic; matches the volume floor in semrush-mcp-cheatsheet.md "Keyword filtering rules"). Keep rows with no volume only if they look like strong question themes (recognizable "how to / what is / why does / can I" prefixes); these often have under-counted long-tail volume.
   • Drop branded-competitor terms (e.g. for Pleasur.AI, drop questions containing candy.ai, ourdream.ai, createporn.com etc. — pull this list from cache/competitors.json if present, otherwise skip the filter).
   • Cap per-seed expansion at 20 results so a 10-seed brand yields ≤ 200 raw Source A candidates before the global cap kicks in.
   • Tag each surviving row with question_subtype = km_question.

4. Source B — PAA mining from top 30 candidates.

   • Sort the existing rows in keyword-ideas.csv by priority_score descending, falling back to traffic_potential descending when priority_score is empty (which is normal pre-Layer-2 — the column is filled by Layer 5).
   • Take the top 30 rows.
   • For each, call mcp__semrush__serp-results and parse serp_features.people_also_ask for the literal PAA strings.
   • Each PAA string becomes a candidate. Drop strings shorter than 4 words (PAA boilerplate), longer than 20 words (long-form questions that don't survive in Google), or containing brand-name placeholders ({, }, etc.).
   • Tag each surviving row with question_subtype = paa.

5. Cap to 100 rows per run. After collecting Source A + Source B, deduplicate-then-cap:

   • First, dedupe within the new-rows pool (case-insensitive, trim whitespace) so two seeds producing the same question collapse to one row.
   • Sort the deduped pool by an internal tier order: paa > km_question, then by volume descending. PAA wins the tie because PAA presence is a direct SERP-feature signal of search intent, whereas Keyword Magic Questions can include long-tail noise.
   • Keep the top 100. Drop the rest. Log the drop count to cache/question-mining-skipped.log for visibility.

6. Merge with existing rows in keyword-ideas.csv. For each capped new row:

   • If the keyword already exists (case-insensitive match on the keyword column): set the existing row's source to both, set question_subtype to both if the existing subtype was already populated (from a prior Layer 1d run with a different subtype), otherwise set to the new subtype. Don't overwrite metric columns (volume, kd, traffic_potential, etc.) — the existing row's values were already enriched by Layer 1b/1c.
   • If the keyword is new: append a fresh row with:
     • keyword, volume, kd, traffic_potential (from Source A; for Source B, traffic_potential may be missing — write empty string)
     • parent_topic (from Source A's first_keyword_group / KSB cluster id; for Source B, run mcp__semrush__keyword-magic-questions with the PAA string as the keyword to enrich, OR write empty string and let Layer 2 enrich during BID — the latter saves quota)
     • intent (from Source A's intents array, joined as comma-list)
     • source = question_mining
     • question_subtype = km_question or paa
     • serp_features = empty (Layer 2 fills this)
     • All BID/AIO/redteam columns and priority_score, notes left empty (Layers 2-5 fill them)

7. Write keyword-ideas.csv atomically (write to .tmp then move). Preserve the original column order; the new question_subtype column lives between source and serp_features (i.e. just after the existing source column, so it's logically grouped with the other source-related metadata).

8. Print a one-line summary: rows from Source A, rows from Source B, merges into existing rows, new rows added, skipped (over the 100 cap), top 5 new question keywords by volume.

Output

Rows appended (or merged) to content-pipeline/0-keywords/keyword-ideas.csv. New column: question_subtype.

Column contract

After Layer 1d, the row contract for keyword-ideas.csv is:

The question_subtype field stays empty for rows that didn't come from Layer 1d (competitor_gap, seed_modifier, aio_gap-only sources). Layer 5 (/keyword-prioritization) doesn't apply a boost on question_subtype directly — question keywords flow through the same scoring, with their lower KD% and higher informational intent already lifting priority_score naturally.

Why the cap is 100

Two reasons:

1. Quota discipline. A 10-seed brand could in theory produce 10 × 20 = 200 Source A rows + 30 Source B rows = 230 candidates. Half of those would be near-duplicates of existing rows or low-volume long-tail noise. 100 is the empirically-supported "where the signal stops mattering" line — past the 100th row, downstream BID rejection rates approach 80%, so the marginal API cost stops paying for itself.
2. Layer 2 throughput. Layer 2 (BID) does ~3 SERP/Domain calls per row. An extra 100 rows is ~300 additional Semrush calls — material but not catastrophic. 200+ would push Layer 2 past the per-run quota window for budget-tier Semrush plans.

If the cap bites repeatedly (logged > 5 times per week to cache/question-mining-skipped.log), revisit the per-seed cap (currently 20) before raising the global cap — the per-seed cap is the right knob for noise-vs-signal trade-off.

Quality checklist

• question_subtype column exists in keyword-ideas.csv after the run
• All Layer 1d new rows have source = question_mining (and question_subtype ∈ {km_question, paa})
• Existing rows that received a Layer 1d match have source = both and question_subtype populated
• No more than 100 new+merged rows added by this run
• PAA strings are 4–20 words; below/above is dropped
• No competitor-branded questions in km_question rows (when cache/competitors.json exists)
• Atomic CSV write — no half-written file on crash

Failure handling

• mcp__semrush__keyword-magic-questions errors for a single seed: log the seed + reason to cache/question-mining-failed.log and skip that seed; continue with the rest. A single seed failing isn't fatal; some seeds reliably produce no question-form variants.
• mcp__semrush__serp-results rate-limit (429) during Source B: persist whatever rows were collected in Source A, write them to keyword-ideas.csv, then exit code 75. The orchestrator handles retry on the next cron cycle (matches the convention in keyword-research-pipeline/SKILL.md).
• seeds.json missing: log to cache/question-mining-failed.log, skip Source A, run Source B alone, exit 0. Layer 1d still adds value from PAA mining even without seed expansion.
• keyword-ideas.csv empty / missing top-30: write the new column header, run Source A only, exit 0. Source B contributes nothing without an existing pool, which is fine — Layer 1d isn't the candidate-pool generator (Layers 1b/1c are).

Layer 1d is not blocking in the same way Layer 0 isn't blocking — the orchestrator continues to Layer 2 even if Layer 1d added zero rows. But unlike Layer 0, an outright crash here propagates because it indicates an MCP/connectivity problem the operator should see.

When question keywords overlap heavily with existing rows

That's fine. The merge logic preserves the richer metadata from Layer 1b/1c (which had competitor_top_position / aio_engines etc.) and just upgrades the source field to both. The question_subtype field gets set so Layer 2's intent classification and Layer 5's prioritization can still see "this keyword is question-shaped" as a signal.

When the merge rate exceeds 70% (logged in the run summary), it usually means either: (a) the brand's category is already well-mined and Layer 1d isn't surfacing fresh territory — fine, the layer is essentially a no-op week; or (b) the seed list is too narrow — escalate to revisiting seeds.json via /seed-modifier-prompt --regen.

Invocation from the master orchestrator

/keyword-research-pipeline calls Layer 1d after Layer 1c (/keyword-aio-gap) and before Layer 2 (/keyword-vet-bid). The orchestrator's brief for this layer is:

You are running Layer 1d at {ROOT}.

Your job: append question-shaped keywords to content-pipeline/0-keywords/keyword-ideas.csv per .claude/skills/keyword-question-mining/SKILL.md. Read the SKILL.md first.

Source A: for each seed in seeds.json, call mcp__semrush__keyword-magic-questions; cap 20 results per seed; tag question_subtype=km_question.
Source B: for top-30 by priority_score (or traffic_potential pre-Layer-2), call mcp__semrush__serp-results, parse serp_features.people_also_ask; tag question_subtype=paa.

Cap 100 new+merged rows total. Existing rows get source=both on overlap.

If serp-results returns 429, persist Source A rows and exit 75.

Return: Source A count, Source B count, merges into existing rows, new rows, skipped (cap), top 5 new question keywords by volume. Under 250 words.

References

• .claude/skills/research/references/semrush-mcp-cheatsheet.md — Keyword Magic filtering rules (volume floor ≥ 10, intent classification, KSB clustering)
• .claude/skills/keyword-research-pipeline/references/semrush-mcp-tool-inventory.md — verify mcp__semrush__keyword-magic-questions and mcp__semrush__serp-results resolve to live tool names
• .claude/skills/keyword-research-pipeline/references/semrush-metric-translation.md — KD% thresholds (Layer 1 floor: ≤ 70), intent-array primacy
• .claude/skills/content-gap-analysis/SKILL.md — column contract for keyword-ideas.csv (Layer 1d preserves and extends it)