# Synthetic Question Generation

> Use this skill when generating new failure-targeting test cases. Documents techniques for producing questions that are diverse, non-trivial, and actually defeat the current pipeline — not paraphrases of existing test cases. Most useful for the red team agent.

- Skill: `allanwessels/synthetic-question-generation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add allanwessels/synthetic-question-generation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/allanwessels/synthetic-question-generation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: AllanWessels (https://skillmd.com/u/allanwessels)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/allanwessels/synthetic-question-generation

---


# Synthetic Question Generation

## When to use

The red team agent uses this skill to generate new test cases. The goal
is *adversarial coverage growth* — finding holes in the test set that
correspond to real holes in the pipeline.

## The trap to avoid

Naive LLM question generation produces variations on a theme. If your
existing test set has 10 questions about pricing, asking an LLM to
"generate more questions" gives you 10 more questions about pricing
phrased slightly differently. You want different *kinds* of questions,
not different *wordings* of the same kind.

## Generation strategies, ranked by adversarial yield

### 1. Multi-hop construction

Pick two random corpus passages. Find what's interesting about each.
Construct a question whose answer requires combining both.

Example: passage A describes the product's free tier limits, passage
B describes pricing in EU vs US. Question: "What's the EU price for
the smallest plan that exceeds free-tier limits?"

This often defeats single-pass retrieval — the system gets one
passage but not both.

### 2. Paraphrase distance maximization

Pick a corpus passage. Identify its key terms. Generate a question
whose answer is the passage but which uses *none* of those terms.

Example: passage says "the system supports horizontal autoscaling
with up to 100 worker pods". Question: "Can the cluster grow when
load spikes?"

Defeats pure vector retrieval when the embedding model leans on
surface tokens.

### 3. Disambiguation pressure

Find two corpus entities with similar names (Product A v1, Product A
v2). Generate a question that's specific to one but could be confused
for the other.

Defeats retrieval when the embedding model collapses similar entities
into the same vector region.

### 4. Negation and scope

"What is NOT supported in the free tier?" "What features are excluded
from the basic plan?" Questions with negation or scope qualifiers
often defeat naive retrieval because the retriever finds passages
about what *is* included.

### 5. Temporal phrases

"What changed in the last release?" "What was the policy before
2024?" Requires both retrieval-time filtering and reasoning over
temporal qualifiers in the corpus.

### 6. Out-of-scope refusal probes

Generate questions whose answers do NOT exist in `/corpus/`. The
pipeline should refuse, not hallucinate. This is a different failure
mode (hallucination) that tests refusal behavior.

## Procedure for the red team

1. Look at `/reports/latest.json` → `by_failure_category`. Find the
   category with the FEWEST current test cases. Target that.

2. Read 5-10 random corpus passages. Note which entities, time
   periods, and topics they cover.

3. Use the strategy that matches the under-tested category. If
   `multi_hop` is underrepresented, use strategy 1. If
   `paraphrase_brittleness` is underrepresented, use strategy 2.

4. Generate 10 candidates. For each, verify the answer is in the
   corpus by citing the exact passage. If you can't cite, throw it
   out.

5. Run each candidate through `scripts/eval_single.py`. Keep only
   the ones that fail (composite < 0.6).

6. Append the kept cases to `/test_cases/generated/<timestamp>.jsonl`.

## Quality bar

Each test case must be:

- **Answerable** — the ground truth is in `/corpus/`, you can point
  to the passage
- **Fair** — a reasonable system *could* answer it given the corpus.
  Trick questions that no system could handle are useless
- **Specific** — has a single correct answer, not five plausible ones
- **Diverse** — meaningfully different from existing cases in
  category, phrasing, and corpus coverage

If you're producing 10 cases that all hit the same passage of the
same document, you're paraphrasing, not testing.