Testing Prompt Changes

HIGR Prompt Testing

Test prompt and skill changes systematically with Hypothesis-Implement-Grade-Rework cycles. Every modification requires testable predictions and evidence-based grading against benchmark data.

Quick Start

1. Write testable hypotheses before editing any prompt or skill
2. Implement the change (delete superseded content)
3. Run against benchmark inputs, compare to known-good output
4. Grade each hypothesis: Good / Bad / Ugly
5. Rework if needed; record results in workspace/higr_log.md

When to Use

• Modifying phase prompts (.github/prompts/phase_*.prompt.md)
• Updating skill SKILL.md or references/
• Changing agent definition (.github/agents/customer_newsletter.agent.md)
• Changing copilot-instructions.md
• Any change that affects newsletter output quality

HIGR Cycle

flowchart LR
    H[Hypothesis] --> I[Implement]
    I --> G[Grade]
    G --> R{Rework?}
    R -->|Bad/Ugly| H
    R -->|Good| D[Done]

Step 1: Hypothesis (Pre-Implementation)

Before editing any prompt or skill, write testable predictions:

Hypothesis Requirements:

• Specific — Not "newsletter will be better"
• Observable — Can see the difference in output
• Falsifiable — Possible to determine if wrong

Example Good Hypotheses:

Step 2: Implement

Make the change. Delete superseded content immediately — no parallel implementations.

Rules:

• No _v2, _new, _enhanced variants
• No backward compatibility code
• Replace existing instructions directly

Step 3: Grade (Post-Implementation)

Run against benchmark data and grade each hypothesis:

Grade Definitions:

Step 4: Rework (If Needed)

For each Bad or Ugly outcome:

Benchmark Testing

Primary Benchmark: December 2025 Cycle

Known-good intermediates for each pipeline phase:

Testing Procedure

1. Invoke the skill with its benchmark input (DATE_RANGE or known-good upstream output)
2. Save output to workspace/
3. Compare against the known-good benchmark file listed above
4. Grade using the comparison dimensions for that skill (see TRAINING_RUNBOOK.md)

Comparison Dimensions by Phase

Secondary Benchmarks (Regression Testing)

After changes stabilize, test against additional cycles:

HIGR Log

Record all HIGR cycles in workspace/higr_log.md:

### HIGR: [Skill] — [Issue] — [Date]

**Hypothesis:** [What should change and why]
**Implement:** [What was changed, file + section]
**Grade:** [Good/Bad/Ugly] — [Evidence]
**Rework:** [If Bad: what was tried next]

Anti-Patterns

Done When

• Hypotheses written (testable, specific, observable)
• Change implemented (old content deleted)
• Benchmark test executed against known-good intermediate
• Each hypothesis graded: Good / Bad / Ugly
• HIGR log updated in workspace/higr_log.md

Converted and distributed by TomeVault — claim your Tome and manage your conversions.