# CLI Eval

> ---

- Skill: `jaccen/cli-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add jaccen/cli-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jaccen/cli-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: jaccen (https://skillmd.com/u/jaccen)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jaccen/cli-eval

---

---
name: cli-eval
description: Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
---
<!-- generated by src/lib/agentSkills/generator.ts; manual edits will be overwritten -->

## Overview

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

## Quick install

```bash
npm install -g AIRoute   # or: npx AIRoute
AIRoute --version
```

## Subcommands

### `eval`

**Example:**

```bash
AIRoute eval
```

### `eval suites`

**Example:**

```bash
AIRoute eval suites
```

### `eval list`

**Example:**

```bash
AIRoute eval list
```

### `eval get <suiteId>`

**Example:**

```bash
AIRoute eval get <suiteId>
```

### `eval create`

**Flags:**

- `--file <path>`

**Example:**

```bash
AIRoute eval create
```

### `eval run <suiteId>`

**Flags:**

- `-m, --model <id>`
- `--combo <name>`
- `--concurrency <n>`
- `--tag <tag>`
- `--watch`

**Example:**

```bash
AIRoute eval run <suiteId>
```

### `eval list`

**Flags:**

- `--suite <id>`
- `--status <s>`
- `--since <ts>`
- `--limit <n>`

**Example:**

```bash
AIRoute eval list
```

### `eval get <runId>`

**Example:**

```bash
AIRoute eval get <runId>
```

### `eval results <runId>`

**Flags:**

- `--failed`

**Example:**

```bash
AIRoute eval results <runId>
```

### `eval cancel <runId>`

**Flags:**

- `--yes`

**Example:**

```bash
AIRoute eval cancel <runId>
```

### `eval scorecard <runId>`

**Example:**

```bash
AIRoute eval scorecard <runId>
```

### `simulate [prompt]`

**Flags:**

- `--file <path>`
- `-m, --model <id>`
- `--combo <name>`
- `--reasoning-effort <level>`
- `--thinking-budget <n>`
- `--explain`

**Example:**

```bash
AIRoute simulate [prompt]
```

<!-- skill:custom-start -->
<!-- Migrated from skills/AIRoute-cli-eval/SKILL.md (preserved curated content) -->

# AIRoute — CLI Evals

Requires the `AIRoute` CLI. See [CLI entry-point skill](https://raw.githubusercontent.com/diegosouzapw/AIRoute/main/skills/AIRoute-cli/SKILL.md) for install + global flags.

## What are evals?

Evals are automated test suites that score LLM outputs against expected answers or rubrics. AIRoute stores suites and run results in its local database.

## Eval suites

```bash
AIRoute eval suites list                       # List all eval suites
AIRoute eval suites list --json                # JSON output

AIRoute eval suites get <suiteId>              # Full suite definition
```

### Create a suite

```bash
AIRoute eval suites create \
  --name "code-quality" \
  --rubric "exact-match" \
  --samples-file ./samples.jsonl                 # JSONL: {input, expected_output}
```

Rubric options: `exact-match`, `contains`, `llm-judge`, `regex`.

`--samples-file` format (one JSON object per line):

```jsonl
{"input": "What is 2+2?", "expected_output": "4"}
{"input": "Translate 'hello' to Spanish", "expected_output": "hola"}
```

## Run an eval

```bash
AIRoute eval suites run <suiteId> \
  --model claude-sonnet-4-6                      # Run suite against a specific model

AIRoute eval suites run <suiteId> \
  --model gpt-4o \
  --watch                                        # Live TUI progress (EvalWatch)
```

The run is asynchronous. Use `--watch` for a live terminal dashboard or poll manually:

```bash
RUN_ID=$(AIRoute eval suites run <suiteId> --model claude-sonnet-4-6 --output json | jq -r '.id')
AIRoute eval get $RUN_ID
```

## Manage runs

```bash
AIRoute eval list                              # List all eval runs
AIRoute eval list --json

AIRoute eval get <runId>                       # Run details (status, model, score)
AIRoute eval results <runId>                   # Per-sample results
AIRoute eval scorecard <runId>                 # Full scorecard with pass/fail per sample
AIRoute eval cancel <runId>                    # Cancel a running eval
```

## Scorecard output

```bash
AIRoute eval scorecard <runId> --output json
```

Response fields per sample:

```json
{
  "id": "sample-1",
  "score": 0.95,
  "passed": true,
  "input": "What is 2+2?",
  "output": "4",
  "expected": "4"
}
```

## Comparing models

Run the same suite against multiple models and compare:

```bash
for MODEL in claude-sonnet-4-6 gpt-4o gemini-2.0-flash; do
  AIRoute eval suites run $SUITE_ID --model $MODEL --output json | jq '{model: .model, score: .score}'
done
```

## CI integration

```bash
# Run and fail CI if score drops below threshold
SCORE=$(AIRoute eval suites run $SUITE_ID --model claude-sonnet-4-6 --output json | jq -r '.score')
python3 -c "import sys; score=float('$SCORE'); sys.exit(0 if score >= 0.90 else 1)"
```

## Errors

- `suites create` fails with `invalid rubric` → use one of: `exact-match`, `contains`, `llm-judge`, `regex`
- `suites run` returns `model not found` → verify model ID with `AIRoute models --search <name>`
- `eval get` shows `status: failed` → check `AIRoute logs --search eval` for error details
- `scorecard` returns empty results → the run may still be `running`; poll `AIRoute eval get <runId>` until `status` is `completed`
<!-- skill:custom-end -->