# Gen Eval

> Run generator-evaluator testing against live services

- Skill: `jankneumann/gen-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add jankneumann/gen-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jankneumann/gen-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: jankneumann (https://skillmd.com/u/jankneumann)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/jankneumann/gen-eval

---


# Gen-Eval

Run the generator-evaluator testing framework against live or local services. Generates test scenarios from interface descriptors, executes them, and evaluates results against expected behavior.

## Arguments

`$ARGUMENTS` - Optional flags:
- `--descriptor <path>` — Path to interface descriptor YAML (auto-detected if omitted)
- `--mode <mode>` (default: `template-only`) — `template-only`, `cli-augmented`, or `sdk-only`
- `--cli-command <cmd>` (default: `claude`) — CLI tool for cli-augmented mode
- `--time-budget <minutes>` (default: `60`) — Time budget for CLI mode
- `--sdk-budget <usd>` — USD budget cap for SDK mode
- `--max-iterations <n>` (default: `1`) — Feedback loop iterations
- `--parallel <n>` (default: `5`) — Concurrent scenario execution
- `--changed-features-ref <git-ref>` — Git ref for change detection
- `--categories <cat1> [cat2 ...]` — Filter to specific categories
- `--report-format <format>` (default: `both`) — `markdown`, `json`, or `both`
- `--output-dir <path>` (default: `.`) — Report output directory
- `--no-services` — Skip service startup/teardown
- `--verbose` — Enable verbose output
- `--openspec-change <change-id>` — OpenSpec change-id whose `### Requirement:` and `#### Scenario:` blocks augment the cli-augmented prompt as additional constraints. Effective only with `--mode cli-augmented`. The change-id MUST match `^[a-zA-Z0-9_-]+$` (alphanumeric, underscore, hyphen only — no path separators or shell metacharacters); invalid values cause exit status 64 before any filesystem walk. When the change directory or its `specs/` subdirectory is missing, gen-eval logs a warning and falls back to descriptor-only generation. Generated `Scenario` objects emitted under this flag include a `source.openspec_scenario` field of the form `openspec/changes/<id>/specs/<rel>.md:<line-start>-<line-end>` so failures can be traced to the originating Requirement.

## Steps

### 1. Auto-Detect Descriptor

If `--descriptor` is not provided, find the nearest descriptor YAML:

```bash
DESCRIPTOR=$(find . -path "*/evaluation/descriptors/*.yaml" -type f 2>/dev/null | head -1)

if [ -z "$DESCRIPTOR" ]; then
  echo "ERROR: No gen-eval descriptor found. Provide --descriptor <path> or create one with /gen-eval-scenario."
  exit 1
fi
echo "Auto-detected descriptor: $DESCRIPTOR"
```

### 2. Detect Project Root and Activate Venv

```bash
# Find the project root (directory containing the descriptor's evaluation/ parent)
PROJECT_ROOT=$(dirname "$(dirname "$(dirname "$(dirname "$DESCRIPTOR")")")")
echo "Project root: $PROJECT_ROOT"

# Activate the project venv
if [ -f "$PROJECT_ROOT/.venv/bin/python" ]; then
  PYTHON="$PROJECT_ROOT/.venv/bin/python"
else
  PYTHON="python3"
fi
```

### 3. Parse Mode and Build Command

Parse `$ARGUMENTS` for mode and flags. Build the CLI command:

```bash
# Defaults
MODE="${MODE:-template-only}"
PARALLEL="${PARALLEL:-5}"
MAX_ITER="${MAX_ITER:-1}"
REPORT_FORMAT="${REPORT_FORMAT:-both}"
OUTPUT_DIR="${OUTPUT_DIR:-.}"

CMD="$PYTHON -m gen_eval --descriptor $DESCRIPTOR --mode $MODE --parallel $PARALLEL --max-iterations $MAX_ITER --report-format $REPORT_FORMAT --output-dir $OUTPUT_DIR"

# Append optional flags from arguments
if [ -n "$TIME_BUDGET" ]; then CMD="$CMD --time-budget $TIME_BUDGET"; fi
if [ -n "$SDK_BUDGET" ]; then CMD="$CMD --sdk-budget $SDK_BUDGET"; fi
if [ -n "$CLI_COMMAND" ]; then CMD="$CMD --cli-command $CLI_COMMAND"; fi
if [ -n "$CHANGED_REF" ]; then CMD="$CMD --changed-features-ref $CHANGED_REF"; fi
if [ -n "$CATEGORIES" ]; then CMD="$CMD --categories $CATEGORIES"; fi
if [ "$NO_SERVICES" = "true" ]; then CMD="$CMD --no-services"; fi
if [ "$VERBOSE" = "true" ]; then CMD="$CMD --verbose"; fi
```

### 4. Run Gen-Eval

Execute from the project root:

```bash
cd "$PROJECT_ROOT"
echo "Running: $CMD"
$CMD
EXIT_CODE=$?
```

### 5. Report Results

After execution, display a summary:

- If reports were generated, read and summarize the markdown report
- Show pass rate, coverage %, and any failures
- If `EXIT_CODE != 0`, highlight failing scenarios and suggest `/gen-eval-scenario` for authoring targeted scenarios

```bash
if [ -f "$OUTPUT_DIR/gen-eval-report.md" ]; then
  echo ""
  echo "=== Gen-Eval Report ==="
  cat "$OUTPUT_DIR/gen-eval-report.md"
fi
```

## Quick Start

The simplest invocation — auto-detects the descriptor and runs template-only:

```bash
/gen-eval
```

With CLI-augmented generation (subscription-covered):

```bash
/gen-eval --mode cli-augmented --time-budget 30
```

Augmenting the cli-augmented prompt with OpenSpec scenarios from an active change:

```bash
/gen-eval --mode cli-augmented --openspec-change my-feature-change-id
```

Against specific categories:

```bash
/gen-eval --categories lock-lifecycle auth-boundary
```

## Integration Points

- **`/validate-feature`**: Gen-eval runs as phase `4b` (between smoke and e2e). Auto-detected when descriptors exist.
- **`/explore-feature`**: Gen-eval report signals (failing interfaces, coverage gaps) feed into feature opportunity ranking.
- **`/gen-eval-scenario`**: Create new scenario YAML files interactively.
- **`make gen-eval`**: Makefile shorthand for the most common invocation.

## Output

- `gen-eval-report.md` — Markdown report with pass/fail summary
- `gen-eval-report.json` — Machine-readable results
- `gen-eval-metrics.json` — Per-scenario metrics for pipeline integration
- Exit code 0 if pass rate meets threshold (default 95%), 1 otherwise