AllanWessels Synthetic Question Generation
When to use
The red team agent uses this skill to generate new test cases. The goal is adversarial coverage growth — finding holes in the test set that correspond to real holes in the pipeline.
The trap to avoid
Naive LLM question generation produces variations on a theme. If your existing test set has 10 questions about pricing, asking an LLM to "generate more questions" gives you 10 more questions about pricing phrased slightly differently. You want different kinds of questions, not different wordings of the same kind.
Generation strategies, ranked by adversarial yield
1. Multi-hop construction
Pick two random corpus passages. Find what's interesting about each. Construct a question whose answer requires combining both.
Example: passage A describes the product's free tier limits, passage B describes pricing in EU vs US. Question: "What's the EU price for the smallest plan that exceeds free-tier limits?"
This often defeats single-pass retrieval — the system gets one passage but not both.
2. Paraphrase distance maximization
Pick a corpus passage. Identify its key terms. Generate a question whose answer is the passage but which uses none of those terms.
Example: passage says "the system supports horizontal autoscaling with up to 100 worker pods". Question: "Can the cluster grow when load spikes?"
Defeats pure vector retrieval when the embedding model leans on surface tokens.
3. Disambiguation pressure
Find two corpus entities with similar names (Product A v1, Product A v2). Generate a question that's specific to one but could be confused for the other.
Defeats retrieval when the embedding model collapses similar entities into the same vector region.
4. Negation and scope
"What is NOT supported in the free tier?" "What features are excluded from the basic plan?" Questions with negation or scope qualifiers often defeat naive retrieval because the retriever finds passages about what is included.
5. Temporal phrases
"What changed in the last release?" "What was the policy before 2024?" Requires both retrieval-time filtering and reasoning over temporal qualifiers in the corpus.
6. Out-of-scope refusal probes
Generate questions whose answers do NOT exist in /corpus/. The
pipeline should refuse, not hallucinate. This is a different failure
mode (hallucination) that tests refusal behavior.
Procedure for the red team
Look at
/reports/latest.json→by_failure_category. Find the category with the FEWEST current test cases. Target that.Read 5-10 random corpus passages. Note which entities, time periods, and topics they cover.
Use the strategy that matches the under-tested category. If
multi_hopis underrepresented, use strategy 1. Ifparaphrase_brittlenessis underrepresented, use strategy 2.Generate 10 candidates. For each, verify the answer is in the corpus by citing the exact passage. If you can't cite, throw it out.
Run each candidate through
scripts/eval_single.py. Keep only the ones that fail (composite < 0.6).Append the kept cases to
/test_cases/generated/<timestamp>.jsonl.
Quality bar
Each test case must be:
- Answerable — the ground truth is in
/corpus/, you can point to the passage - Fair — a reasonable system could answer it given the corpus. Trick questions that no system could handle are useless
- Specific — has a single correct answer, not five plausible ones
- Diverse — meaningfully different from existing cases in category, phrasing, and corpus coverage
If you're producing 10 cases that all hit the same passage of the same document, you're paraphrasing, not testing.