# Prompt Lab > Iterate on LLM prompts with structured evaluation and self-correction. Test prompts against ground truth, compare models, track version history. Self-correction loop sends invalid outputs back to LLM for fixing. Supports both taxonomy classification and QRA (Question-Reasoning-Answer) generation. - Skill: `majiayu000/prompt-lab-2` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add majiayu000/prompt-lab-2` - Raw SKILL.md: https://api.skillmd.com/api/skills/majiayu000/prompt-lab-2/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: majiayu000 (https://skillmd.com/u/majiayu000) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/majiayu000/prompt-lab-2 --- # Prompt Lab Systematic prompt engineering with ground truth evaluation and **self-correction loop**. ## Key Feature: Self-Correction Loop Unlike simple validation that silently filters invalid outputs, this skill: 1. **Presents vocabulary in prompt** - LLM knows valid options upfront 2. **Validates response** - Pydantic catches invalid tags 3. **Sends correction back to LLM** - "You used invalid tags X. Valid options are Y. Please fix." 4. **Tracks correction rounds** - Metrics show how often LLM needed help This gives the model a chance to self-correct rather than silently failing. ## Quick Start ```bash cd /home/graham/workspace/experiments/pi-mono/.pi/skills/prompt-lab # Find the smallest model that works (NEW!) ./run.sh find-minimum --ground-truth queryspec.json --threshold 0.80 # Run evaluation with self-correction enabled (default) ./run.sh eval --prompt taxonomy_v1 --model deepseek # Compare multiple models on same prompt ./run.sh compare --prompt taxonomy_v1 --models "deepseek,gpt-4o" # View evaluation history ./run.sh history --prompt taxonomy_v1 ``` ## Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ Stage 1: LLM Call with Vocabulary in Prompt │ │ - Vocabulary clearly defined (Tier 0, Tier 1 tags) │ │ - JSON response format enforced │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ Stage 2: Pydantic Validation │ │ - Parse JSON response │ │ - Detect invalid/hallucinated tags │ │ - Track rejected tags for metrics │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ Stage 3: Self-Correction Loop (if invalid tags detected) │ │ - Send assistant message: "Invalid tags: X" │ │ - Ask LLM to fix: "Valid options are: Y" │ │ - Retry up to N times (default: 2) │ │ - Track correction rounds in metrics │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ Stage 4: Evaluation Metrics │ │ - Precision/Recall/F1 vs ground truth │ │ - Correction rounds needed │ │ - Correction success rate │ └─────────────────────────────────────────────────────────────┘ ``` ## Commands ### find-minimum - Find Smallest Accurate Model (NEW!) Automatically test models from smallest to largest, stopping at the first model that meets your accuracy threshold. Supports both Chutes API and local Ollama. ```bash # Find smallest model for QuerySpec with 80% accuracy threshold ./run.sh find-minimum --ground-truth queryspec.json --threshold 0.80 # Prefer local Ollama models (default) ./run.sh find-minimum -g queryspec.json -t 0.80 --prefer-local # Test Chutes API models instead ./run.sh find-minimum -g queryspec.json -t 0.80 --no-prefer-local # Options: # --ground-truth, -g Ground truth JSON file (required) # --threshold, -t Minimum accuracy threshold (default: 0.80) # --prompt, -p Optional prompt file # --prefer-local Prefer local Ollama (default: true) # --max-models Max models to test (default: 10) # --verbose, -v Show per-case details ``` **Model Sources:** - **Ollama (local)**: qwen3:1.7b, qwen2.5-coder:7b, qwen3:8b - **Chutes (API)**: Qwen2.5-Coder-32B, Qwen3-32B, DeepSeek-V3, etc. **Output:** - Sorted table of all tested models with JSON% and Action% - Recommended model with size and provider - Results saved to `results/find_minimum_*.json` ### eval - Run Evaluation ```bash ./run.sh eval --prompt taxonomy_v1 --model deepseek # Options: # --prompt NAME Prompt version to test # --model NAME Model to use (deepseek, gpt-4o, etc.) # --cases N Number of test cases (default: all) # --max-corrections N Max self-correction rounds (default: 2) # --no-correction Disable self-correction loop # --task-name NAME Task-monitor task name for quality gate # --verbose Show per-case details ``` ### compare - Compare Models ```bash ./run.sh compare --prompt taxonomy_v1 --models "deepseek,gpt-4o" # Output: # | Model | F1 | Corrections | Time | # |----------|-------|-------------|-------| # | deepseek | 0.90 | 3 | 2.3s | # | gpt-4o | 0.93 | 1 | 1.1s | ``` ### extract-prompts - Extract Prompts from Python ```bash ./run.sh extract-prompts --file /path/to/12_qra.py --output prompts/ ``` ### test-sparta - End-to-End SPARTA QRA Test ```bash ./run.sh test-sparta --db-path /path/to/sparta.duckdb --cases 100 ``` ## Advanced Usage ### test-sparta Options | Option | Default | Description | |--------|---------|-------------| | `--phase <0\|1>` | 0 | 0=Relationships (Technique->Control), 1=Simple Control QRA | | `--threshold ` | 0.85 | Citation grounding threshold | | `--min-anchoring` | 0.995 | Entity anchoring threshold (99.5% for LLM non-determinism) | | `--min-ambiguity` | 0.995 | Ambiguity gate threshold (99.5% for LLM non-determinism) | | `--min-grounding` | 0.90 | Citation grounding threshold | | `--json` | false | Output structured JSON summary | | `--json-stream` | false | Output NDJSON per case (streaming progress) | | `--task-monitor/--no-task-monitor` | true | Enable/disable task-monitor integration | | `--converge` | false | Enable iterative convergence mode | ### Task-Monitor Integration prompt-lab integrates with the centralized task-monitor for live progress tracking: ```bash # Run test with task-monitor (enabled by default) ./run.sh test-sparta --cases 100 # View progress in task-monitor TUI cd ~/.pi/skills/task-monitor uv run python monitor.py tui --filter prompt-lab # Or check state file directly cat /path/to/prompt-lab/prompt_lab_task_state.json | jq ``` State file schema: ```json { "completed": 50, "total": 100, "progress_pct": 50.0, "gates": { "ambiguity": {"passed": 50, "failed": 0, "rate": 1.0}, "anchoring": {"passed": 49, "failed": 1, "rate": 0.98}, "grounding": {"passed": 48, "failed": 2, "rate": 0.96} }, "failures": [...], "status": "running" } ``` ### NDJSON Streaming Output For long-running tests, use `--json-stream` to output one JSON object per line: ```bash ./run.sh test-sparta --cases 500 --json-stream | tee results.jsonl # Each line: # {"case_id": "control_T1110", "status": "pass", "qra_count": 5, "metrics": {...}, "latency_ms": 1234.5} ``` This enables: - Real-time progress monitoring via `tail -f results.jsonl | jq` - Integration with streaming parsers - Resume from partial runs ### Validation Features - **Ambiguity Gate**: Checks for sufficient length and context keyword usage. - **Entity Anchoring**: Verifies questions explicitly name the subject entities. - **Citation Grounding**: Verifies answers are derived verbatim from source text. ### history - View History ```bash ./run.sh history --prompt taxonomy_v1 # Shows all evaluation runs with scores over time ``` ### analyze - Analyze Past Results ```bash ./run.sh analyze --prompt taxonomy_v1 # Analyzes error patterns across all previous evaluations: # - Most common invalid tags # - Cases needing correction # - Performance trend over time # - Suggests improvements based on patterns ``` ### optimize - LLM-Powered Prompt Optimization ```bash ./run.sh optimize --prompt taxonomy_v1 # Uses LLM to analyze error cases and suggest prompt improvements: # - Reviews cases with low F1 scores # - Identifies ambiguous tag definitions # - Generates revised prompt sections # - Saves suggestions for review ``` ## Self-Correction Prompt When invalid tags are detected, this correction message is sent back to the LLM: ``` Your response contained invalid tags that are not in the allowed vocabulary. Invalid tags you used: {rejected_tags} Valid conceptual tags (Tier 0): Corruption, Fragility, Loyalty, Precision, Resilience, Stealth Valid tactical tags (Tier 1): Detect, Evade, Exploit, Harden, Isolate, Model, Persist, Restore Please correct your response. Return ONLY valid JSON with tags from the allowed vocabulary above. Do NOT invent new categories. Only use the exact tag names listed. ``` ## Quality Gates Evaluation enforces quality gates: - **F1 >= 0.8** - Must achieve 80% F1 score against ground truth - **Correction Success >= 90%** - Self-correction must succeed 90% of the time If quality gates fail, exit code is 1 (for CI/CD integration). ## Task-Monitor Integration ```bash # Notify task-monitor of evaluation result ./run.sh eval --prompt taxonomy_v1 --task-name "prompt-validation" # Task-monitor receives: # - Task name # - Pass/fail status # - Metrics (F1, correction rounds, rejected tags) ``` ## Directory Structure ``` prompt-lab/ ├── SKILL.md ├── run.sh ├── prompt_lab.py # Main CLI with self-correction ├── ground_truth/ │ └── taxonomy.json # Test cases with expected outputs ├── prompts/ │ ├── taxonomy_v1.txt # Prompt version 1 │ └── taxonomy_v2.txt # Prompt version 2 ├── results/ │ └── taxonomy_v1_deepseek_20260128.json # Evaluation results ├── models.json # Model configurations └── pyproject.toml ``` ## Metrics - **F1**: Combined precision/recall across all tag types - **Conceptual Precision/Recall**: Tier 0 tag accuracy - **Tactical Precision/Recall**: Tier 1 tag accuracy - **Correction Rounds**: Total self-correction attempts across all cases - **Correction Success Rate**: Percentage of cases where self-correction worked - **Rejected Tags**: Total hallucinated tags caught (before and after correction) ## Model Configuration Models are configured in `models.json`: ```json { "deepseek": { "provider": "chutes", "model": "deepseek-ai/DeepSeek-V3-0324-TEE", "api_base": "$CHUTES_API_BASE", "api_key": "$CHUTES_API_KEY" }, "gpt-4o": { "provider": "openai", "model": "gpt-4o", "api_key": "$OPENAI_API_KEY" } } ``` ## Use Cases 1. **Taxonomy Prompt Iteration**: Improve SPARTA bridge tag extraction 2. **QRA Prompt Testing**: Test question-reasoning-answer extraction with citation grounding 3. **Classification Tasks**: Any structured output with controlled vocabulary 4. **Model Selection**: Compare providers for quality/cost tradeoffs 5. **Self-Correction Analysis**: Understand which models need more guidance ## QRA Evaluation (Roadmap) Prompt-lab currently supports taxonomy classification. Based on real-world usage in SPARTA QRA generation, the following enhancements are planned: ### Current Limitations - Only evaluates conceptual/tactical tag F1 scores - No citation grounding validation - No multi-item response support (QRA generates 4-7 items per input) - No hallucination detection - No question type diversity metrics ### Planned Features **QRA Evaluation Mode:** ```bash ./run.sh eval-qra --prompt qra_v2 --model deepseek --cases 5 ``` **New Metrics:** - Citation grounding rate (% of citations that match source text verbatim) - Question type distribution (simple/medium/complex/reversal_curse) - Persona distribution (lay_person/project_manager/cybersecurity_expert) - Hallucination count (citations not found in source) - Duplicate answer detection - Confidence distribution (strong/partial/inference) **Citation Grounding Validation:** Uses rapidfuzz to verify every citation is an exact verbatim excerpt from source text: ```python def validate_citation(citation: str, source_text: str, threshold: float = 0.85) -> bool: from rapidfuzz import fuzz return fuzz.partial_ratio(citation.lower(), source_text.lower()) >= threshold * 100 ``` ### Key Learnings from SPARTA QRA Generation **What Worked:** | Technique | Outcome | | ---------------------------------------------- | ---------------------------------------------------- | | "Think like a LEGAL LLM - cite precedent" | Enforced verbatim citation behavior | | "STRICT GROUNDING - if not in text, don't add" | Eliminated hallucinated mitigation advice | | "Generate ALL reasonable questions" | Increased diversity from 1-2 to 4-7 QRAs per input | | "Each extracts DIFFERENT information" | Reduced duplicate answers | | "EXACT verbatim excerpt" for citations | Stopped control ID citations, enforced text snippets | **What Didn't Work:** | Anti-Pattern | Problem | | ---------------------------------- | --------------------------------------- | | Template examples in prompts | Caused repetitive outputs | | Generic "be grounded" instructions | Too vague, still hallucinated | | "Generate ONE per type" | Limited coverage of source material | | No deduplication guidance | Repeated same answer in different words | | "Citation: [CONTROL_ID]" | Generated IDs instead of text excerpts | ### Refined QRA Prompt Template The following prompt pattern generates properly grounded QRAs: ``` You are a space-based cybersecurity expert generating Question-Reasoning-Answer pairs for SPARTA. Think like a LEGAL LLM: every claim MUST cite precedent from the provided text. STRICT GROUNDING RULE: If information is NOT in the provided text, you CANNOT include it in your answer. Do NOT add external knowledge, mitigation advice, or implications not directly stated. Return JSON: {"items": [{"question": "...", "question_type": "simple|medium|complex|reversal_curse", "questioner_persona": "lay_person|project_manager|cybersecurity_expert", "reasoning": "...", "answer": "...", "citations": ["EXACT verbatim excerpt from text"], "confidence": "strong|partial", "conceptual_tags": [], "tactical_tags": []}]} GENERATE ALL REASONABLE NON-DUPLICATE QUESTIONS: - Simple, Medium, Complex levels + Reversal curse where applicable - Each extracts DIFFERENT information from the text - NEVER add information not in the source text TAXONOMY: C=[Corruption,Fragility,Loyalty,Precision,Resilience,Stealth], T=[Detect,Evade,Exploit,Harden,Isolate,Model,Persist,Restore] ``` ### Implementation Status - [x] Create [`citation_validator.py`](citation_validator.py) with rapidfuzz - [x] Add `QRAEvalSummary`, `QRAGroundedTestCase`, `QRAGroundedResult` dataclasses to [`evaluation.py`](evaluation.py) - [x] Create [`ground_truth/qra_grounded.json`](ground_truth/qra_grounded.json) schema - [x] Create example prompt [`prompts/qra_grounded_v1.txt`](prompts/qra_grounded_v1.txt) - [x] Add `load_qra_grounded_truth()` function to [`evaluation.py`](evaluation.py) - [x] Add `eval-qra` command to [`prompt_lab.py`](prompt_lab.py) CLI - [x] [`run.sh`](run.sh) automatically supports eval-qra (passes all args to CLI) - [x] QRA-specific metrics in evaluation output ### Files Added - [`citation_validator.py`](citation_validator.py) - Citation grounding validation with rapidfuzz fuzzy matching - [`ground_truth/qra_grounded.json`](ground_truth/qra_grounded.json) - Schema with source_text and citation requirements - [`prompts/qra_grounded_v1.txt`](prompts/qra_grounded_v1.txt) - Working prompt template from SPARTA production use ### Usage The `eval-qra` command is now available: ```bash # Run QRA evaluation with default settings ./run.sh eval-qra --prompt qra_grounded_v1 --model deepseek # With custom threshold and verbose output ./run.sh eval-qra --prompt qra_grounded_v1 --model deepseek --threshold 0.90 --verbose # Test on limited cases ./run.sh eval-qra --prompt qra_grounded_v1 --model deepseek --cases 2 ``` **Output includes:** - Total QRAs generated - Average QRAs per input - Citation grounding rate - Hallucination count - Duplicate detection - Question type coverage - Average latency **Quality Gates:** - Citation grounding ≥ 85% - Zero hallucinations For details, see: [`../../.agent/inbox/memory_prompt-lab-qra-improvements_20260131_125541.md`](../../.agent/inbox/memory_prompt-lab-qra-improvements_20260131_125541.md)