# AI Eval Scorer

> Take an AI feature spec and produce a full eval harness - extracted behaviors, golden + edge + adversarial test cases, a judge rubric, and a scorecard. Use before shipping any LLM-powered feature, when "it works on my examples" isn't enough, or when stakeholders ask "how do you know it's good". Triggers on phrases like "build evals for this feature", "score this AI feature", "test cases for LLM feature", "eval rubric", "AI quality scorecard", "оцени AI-фичу", "сделай эвалы".

- Skill: `euroconic/ai-eval-scorer` (Agent Skill)
- Install (CLI): `npx skillmds@latest add euroconic/ai-eval-scorer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/euroconic/ai-eval-scorer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: euroconic (https://skillmd.com/u/euroconic)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/euroconic/ai-eval-scorer

---


# AI Eval Scorer

## Purpose

Most AI features ship with vibes-based quality. The PM tries 5 examples, says "looks good", and ships. Two weeks later, customers find the broken cases.

This skill replaces vibes with a structured eval harness: explicit behaviors the feature should support, test cases per behavior (golden, edge, adversarial), a judge rubric, and a scorecard you can rerun on every model change.

This is the **signature skill** for PMs who want to ship AI features with the same rigor backend teams use for APIs.

## When to use

- Before shipping any LLM-powered feature to users
- When migrating models (Sonnet 4.6 → Sonnet 4.7, OpenAI → Claude) and need a regression check
- When stakeholders ask "how do you know it's good"
- For features where quality is the entire product (chatbot, summarizer, generator)
- For prompt iterations: compare old vs new prompt against the same suite

## When NOT to use

- For non-AI features (use regular acceptance criteria)
- For one-off internal prototypes nobody depends on
- For features where the "right answer" is genuinely unknowable (creative ideation, exploratory chat)
- Without a real spec - garbage in, useless evals out

## Inputs

Required:
- Feature spec or natural-language description of what the feature does
- Examples of expected behavior if available

Strongly improves quality:
- Target user and use case
- Known edge cases the team is already worried about
- Constraints (length, tone, formatting, prohibited outputs)
- Comparable existing features (for behavioral reference)

## Process

### Phase 1: Extract behaviors

Read the spec and pull out the discrete behaviors the feature is claiming:
- Functional behaviors ("summarizes the input in under 200 words")
- Stylistic behaviors ("matches the brand voice")
- Safety behaviors ("refuses to give medical advice")
- Constraints ("never outputs more than 3 paragraphs")

Each behavior is a testable claim. If a behavior cannot be tested by inspecting the output, mark it untestable and flag for spec revision.

### Phase 2: Generate test cases

For each behavior, generate three case types:

**Golden cases** (5-10 per behavior)
- Clear, representative inputs
- The "happy path"
- These define what good looks like

**Edge cases** (5-10 per behavior)
- Boundary conditions (very short, very long, mixed languages, ambiguous intent)
- Unusual but valid inputs
- These catch the failures that survive into prod

**Adversarial cases** (3-5 per behavior)
- Prompt injection attempts
- Inputs designed to elicit prohibited behavior
- Edge cases the team would lose money on
- These catch the failures that show up in headlines

Each case has: input, expected behavior, scoring criteria, severity if failed.

### Phase 3: Build the judge rubric

For each case, define how to score the model's output:

- **Binary criteria** ("Did it refuse the medical question? Y/N")
- **Scaled criteria** (1-5) with explicit anchors:
  - 1: catastrophic failure
  - 3: acceptable but not great
  - 5: ideal output
- **Reasoning required**: judge must explain the score, not just give a number

Rubric is part of the skill output. The user takes it to a judge model (Claude, GPT-4, human reviewers, or a panel).

### Phase 4: Run the cases (optional, if model access available)

If able to execute:
- Run each case through the feature
- Score with an LLM judge using the rubric
- Aggregate per behavior and overall

If not able to execute:
- Output the runnable suite as a JSONL or markdown table
- Hand off to the eng team to wire into CI

### Phase 5: Scorecard

```
# Eval Scorecard: [Feature]

## Overall: X / Y cases passed (Z% golden, W% edge, V% adversarial)

## Per-behavior scores
| Behavior | Golden | Edge | Adversarial | Verdict |

## Failures by severity
- Critical (ship-blocking)
- Major (must-fix before scale)
- Minor (track, fix in next iteration)

## Qualitative observations
[3-5 themes the judge noticed across cases]

## Recommendations
- Ship / Iterate / Block
- Specific prompt or model changes to try
- Cases to add to regression suite
```

## Output

The skill produces three artifacts:

1. **Behaviors list** (markdown)
2. **Test case suite** (markdown table or JSONL, runnable)
3. **Judge rubric** (markdown, model-agnostic)

Plus the scorecard if cases were actually run.

## Common failure modes to avoid

- **Vague behaviors**: "responds helpfully" is not testable. "Answers the user's question within 3 sentences" is.
- **No adversarial cases**: the team will find them, then customers will. Better here.
- **Hidden rubric**: if the judge's reasoning isn't surfaced, you can't trust the score.
- **Scoring once and forgetting**: this suite is a regression net, rerun it on every model change.
- **Skipping the "untestable" flag**: if the spec has untestable claims, the spec is broken.

## Why this skill exists

AI quality is the new frontier for PMs and most don't have a process for it yet. The teams shipping good AI products treat evals like unit tests: boring, mandatory, and the reason nothing catches fire on launch day.

This skill is the PM-side starting kit. Engineers will build the harness; PMs need to bring the rubric.