Eval

Eval — Quality & Regression Gate

CRITICAL: Run HARNESS_DIR=$(epic path) first. Never use .harness/ in the project directory.

When to Trigger

• Before /ship creates a PR (automatic if eval.yaml exists)
• After /go completes a feature
• On explicit /eval command
• When user mentions "regression", "baseline", "eval suite", "quality check"
• CI: make eval or epic eval --json

Execution Modes

4 dimensions run in parallel where possible:

1. eval:correctness — Test pass rate, mutation score, assertion density
2. eval:performance — Throughput, latency, memory (opt-in)
3. eval:quality — Lint, code quality, LLM-as-judge
4. eval:regression — Baseline comparison, score deltas

Process

Step 0: Prerequisites

HARNESS_DIR=$(epic path)

If $HARNESS_DIR/eval/eval.yaml does not exist, run scaffold:

epic eval --init

Read the config:

cat $HARNESS_DIR/eval/eval.yaml

Step 0.5: Scaffold benchmarks (when no benchmark infrastructure exists)

If eval.yaml has benchmarks: [] and no benchmark files are found in the project:

1. Generate stub files using the CLI:

   epic eval --scaffold
   

   Supported stacks (auto-detected from project markers):

   Stack	Detected by	Generated file	Output format
   Rust	Cargo.toml	benches/eval_harness.rs	criterion (exit code)
   Python	pyproject.toml / setup.py	benchmarks/eval_runner.py	JSON composite
   TypeScript	tsconfig.json	benchmarks/eval.ts	JSON composite
   Node.js	package.json	benchmarks/eval.mjs	JSON composite
   Go	go.mod	benchmarks/eval_test.go	JSON composite
   Java	pom.xml / build.gradle	benchmarks/EvalBenchmark.java	exit code
   Kotlin	build.gradle.kts	benchmarks/EvalBenchmark.kt	exit code
   Ruby	Gemfile	benchmarks/eval_benchmark.rb	JSON composite
   PHP	composer.json	benchmarks/eval_benchmark.php	JSON composite
   C#	*.csproj / *.sln	Benchmarks/EvalBenchmark.cs	JSON composite
   Swift	Package.swift	benchmarks/EvalBenchmark.swift	JSON composite
   Elixir	mix.exs	benchmarks/eval_benchmark.exs	JSON composite
   C++	CMakeLists.txt	benchmarks/eval_benchmark.cpp	exit code

2. Customize the generated file — every file has # TODO / // TODO markers:

   • Replace placeholder logic with calls to your actual domain functions
   • Adjust the composite score weights to reflect your domain priorities
   • For precision/recall benchmarks: wire in your real test set and model

3. If --scaffold can't generate a useful stub (domain too complex, custom evaluation logic needed), generate a custom benchmark with LLM assistance:

   • Read the project's main source files to understand the domain
   • Identify the 2–3 most critical quality signals (latency, accuracy, throughput, precision/recall)
   • Write a benchmark that measures those signals and outputs {"composite": 0.0–1.0, ...}
   • Save to benchmarks/eval_runner.{ext} matching the project language

4. Wire into eval.yaml:

   benchmarks:
     - name: eval_runner
       command: python3 benchmarks/eval_runner.py full
       result_type: composite   # parse composite field from JSON stdout
   

   Use result_type: exit_code for frameworks (criterion, JMH, BenchmarkDotNet) that manage their own output.

Step 1: Run Rust CLI

Execute the structured evaluation via the Rust binary:

epic eval --json

This runs all enabled dimensions and outputs a JSON result. Capture the output.

If the CLI reports llm_judge: SKIPPED (no LLM available in CLI mode), proceed to Step 2 for LLM-as-judge. Otherwise, skip to Step 3.

Step 2: LLM-as-Judge (when llm_judge enabled)

If the quality dimension has llm_judge: true and CLI marked it SKIPPED:

1. Sample 3–5 changed files from the current branch:

   git diff --name-only $(git merge-base HEAD main)
   

2. For each sampled file, evaluate on a 1-10 rubric:

   • Readability (naming, structure, flow)
   • Correctness (logic, edge cases, error handling)
   • DRY (no unjustified duplication)
   • Security (no obvious vulnerabilities)

3. Average scores across files. Map to 0.0–1.0 scale.

4. Record results alongside CLI output.

Step 3: Load Baseline

cat $HARNESS_DIR/eval/baselines/latest.json

If no baseline exists, the current run BECOMES the first baseline. Save it:

epic eval --baseline-update

Report: "First baseline established. Future runs will compare against this."

Step 4: Synthesize Report

Combine CLI output + LLM-as-judge results into a single report:

## Eval Report
- Branch: {branch}
- Commit: {commit_short}

### Correctness: [PASS/WARN/FAIL] — score: {score}
- Tests: {passed}/{total} passing ({pass_rate}%)
- Mutation score: {mutation_score}% (if enabled)
- Delta vs baseline: {+/-delta}

### Performance: [PASS/WARN/FAIL] — score: {score} (if enabled)
- Avg latency: {latency}ms (delta: {+/-delta})
- Throughput: {throughput} (delta: {+/-delta})

### Quality: [PASS/WARN/FAIL] — score: {score}
- Lint errors: {count}
- LLM judge: {score}/10 (if enabled)

### Regression: [PASS/FAIL]
| Dimension | Baseline | Current | Delta | Verdict |
|-----------|----------|---------|-------|---------|
| correctness | {prev} | {cur} | {delta} | {pass/fail} |
| quality | {prev} | {cur} | {delta} | {pass/fail} |

### Overall: [PASS/WARN/FAIL] — {overall_score}

Step 5: Act

• All PASS + no regression: "Eval passed. Run /ship to create a PR."
• WARN: Show warnings. Ask whether to fix before shipping.
• FAIL or regression detected: List each failure with fix hint. "Fix with /go, then re-run /eval."

Step 6: Save Results

epic eval --baseline-update  # if user approves this as new baseline

Results auto-saved to $HARNESS_DIR/eval/results/EVAL-{timestamp}.json.

Anti-Rationalization

Evidence Required

• epic eval --json output captured (all enabled dimensions scored)
• Baseline comparison performed (or first baseline established)
• Each dimension has PASS/WARN/FAIL verdict
• No dimension regressed beyond threshold (or explicit user override)
• Results saved to $HARNESS_DIR/eval/results/
• LLM-as-judge scores recorded (if enabled)

Red Flags

• Reporting PASS without actual epic eval output
• Skipping regression comparison "because it's the first run" (first run should ESTABLISH baseline)
• Reporting PASS with 0 test coverage
• Ignoring mutation score drops >5%
• Marking eval PASS when any dimension below minimum threshold
• Running eval on main branch instead of feature branch