# Evals

> Evaluation skill for AI output: apply measurable criteria to text/code/ research/policy. Supports four eval types: rubric-based (1-5 scale), pairwise (A vs B), checklist (binary), and red-team pass (uses thinking-redteam). Output: score + reasoning + improvement suggestions. [WHAT] Structured way to answer "is this good enough?". Uses domain rubrics: chronicle quality, memo rigor, academic precision, OSINT credibility. Eval is a feedback mechanism, not a gate. [WHEN] Use when: evaluate, score, judge, "is this good enough?", "rate this", quality check, compare A vs B. NOT for: fact-checking (use fact-check), academic peer review (use academic-opponent). [LANGUAGE] English and other languages; matches input.

- Skill: `carlheath/evals` (Agent Skill)
- Install (CLI): `npx skillmds@latest add carlheath/evals`
- Raw SKILL.md: https://api.skillmd.com/api/skills/carlheath/evals/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: carlheath (https://skillmd.com/u/carlheath)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/carlheath/evals

---


# Evals

**Role:** structured evaluation of AI output.

## Four eval types

### 1. Rubric-based (1-5 scale)

Define 3-7 dimensions relevant to the task, score per dimension, justify per score.

### 2. Pairwise (A vs B)

Two versions of the same output. Winner per dimension. Total winner.

### 3. Checklist (binary)

List of must-haves. Yes/no per item. Threshold (e.g. 8/10) = pass.

### 4. Red-team pass

Delegate to the thinking-redteam skill. Pass if <3 weaknesses at Medium or above.

## Domain rubrics

### Chronicle (chronicle skill output)

| Dimension | 1 | 3 | 5 |
|-----------|---|---|---|
| Voice fidelity | Generic AI | Approximates | Distinct author voice |
| Anti-AI patterns | Em-dashes + "not X. It is Y" | Few | None |
| Narrative arc | Linear summary | Attempts arc | Opening-build-landing |
| Concreteness | Abstract | Some examples | Specific numbers, names, dates |
| Closing | "In summary..." | OK | Action call, sharp |

### Memo (policy-brief output)

| Dimension | 1-5 |
|-----------|-----|
| Administrative tone | ... |
| Evidence basis | ... |
| Structural clarity | ... |
| Recommendation sharpness | ... |
| Recipient adaptation | ... |

### Academic (academic output)

| Dimension | 1-5 |
|-----------|-----|
| IMRAD discipline | ... |
| Source criticism | ... |
| Argument chain | ... |
| Terminological precision | ... |
| Original contribution | ... |

### OSINT (person-osint, osint-network output)

| Dimension | 1-5 |
|-----------|-----|
| Source credibility | ... |
| Verification trail | ... |
| Threat-actor perspective | ... |
| Network depth | ... |
| Security-risk handling | ... |

## Output format

```markdown
## Eval: [output name]

### Rubric: [chronicle / memo / academic / OSINT / custom]

| Dimension | Score | Reasoning |
|-----------|-------|-----------|
| [Dim 1] | 4/5 | [Specific observation] |
| [Dim 2] | 2/5 | [What's missing] |
| ... | ... | ... |

**Total: X / Y**

### Top 3 improvement points
1. [Concrete action]
2. [...]
3. [...]

### Pass/Fail
[With given threshold, or pass if >70%]
```

## Auto-eval in ALGORITHM mode

In the ALGORITHM VERIFY phase: run the eval rubric automatically on produced output. If below threshold, iterate. The user sees the eval result, not just the final product.

## Version history

- v3.0 (2026-05-02): initial public release.