# Eval

> View and manage the evaluation harness — session metrics, baselines, trend analysis, and regression detection. The feedback loop that tells the system whether improvements actually improved things.

- Skill: `kastalien-research/eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add kastalien-research/eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kastalien-research/eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: kastalien-research (https://skillmd.com/u/kastalien-research)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/kastalien-research/eval

---


Evaluation harness: $ARGUMENTS

## Commands

Parse the first word of $ARGUMENTS to determine the command:

### `metrics` — Show current session metrics
Collect and display metrics for the current session:

1. Count commits: `git log --oneline --since="today" | wc -l`
2. Count test results: check for recent vitest output or `.eval/metrics/` entries
3. Token usage: check LangSmith state file if available
5. Pattern usage: check `.dgm/fitness.json` for patterns used this session
6. Session duration: check session start time from logs

Display as:
```
## Current Session Metrics

| Metric | Value | Baseline | Delta |
|--------|-------|----------|-------|
| Commits | 5 | 3.2 avg | +56% |
| Tests passing | 42/42 | 40/42 | +2 |

| Files changed | 12 | 8.5 avg | +41% |
| Patterns used | 7 | 5.3 avg | +32% |
```

### `baseline` — Set or update baselines
1. Read the last N session metric snapshots from `.eval/metrics/`
2. Calculate averages for each metric
3. Write to `.eval/baselines.json`
4. Report what changed

### `compare` — Compare sessions
Usage: `compare --last N` or `compare --session <id>`

1. Load metric snapshots from `.eval/metrics/`
2. Compare against baselines
3. Highlight regressions (metric dropped >10% below baseline)
4. Highlight improvements (metric improved >10% above baseline)

### `report` — Generate weekly evaluation report
1. Load all metrics from the past 7 days
2. Calculate trends (improving, stable, declining)
3. Identify top improvements and top regressions
4. Generate recommendations based on trends

### `capture` — Capture current session metrics
Write a metric snapshot to `.eval/metrics/session-{timestamp}.json`:

```json
{
  "session_id": "<session id>",
  "timestamp": "<ISO 8601>",
  "branch": "<git branch>",
  "metrics": {
    "commits": 0,
    "tests_total": 0,
    "tests_passing": 0,
    "files_changed": 0,
    "patterns_referenced": 0,
    "assumptions_verified": 0,
    "escalations": 0,
    "spiral_detections": 0
  },
  "qualitative": {
    "session_focus": "<what the session was about>",
    "memory_usefulness": 0,
    "knowledge_gaps_found": []
  }
}
```

## Notes

- If `.eval/baselines.json` doesn't exist, skip baseline comparisons and suggest running `baseline`
- Metric collection should be best-effort — missing data is noted, not an error
- Regressions trigger a structured escalation suggestion (not automatic action)