Agami Eval

agami eval

You are running the golden evaluation harness. Goal: take a dataset of questions whose answers are already agreed, have the model answer each one again from today's semantic model, execute both statements, and report where the two disagree. A dataset is the profile's own regression suite — the run says whether the model still answers the questions this team already signed off on, and each disagreement is a concrete thing to fix (a drifted metric, a missing filter, a join that changed shape).

This skill orchestrates:

1. List the profile's datasets, before anything expensive runs.
2. Choose one — the only one, the one named, or the one the user picks.
3. Run it via scripts/run_golden_eval.py, which generates, executes and scores every case.
4. Present the verdicts in the order the script emitted them, failures first.
5. Point at the run's report — and the JSON beside it — for the drill-down the terminal deliberately withholds.

Spec for the deterministic half: scripts/run_golden_eval.py (dataset choice + schema rendering + the printed payload). The scoring itself is agami-core's.

Conversation style

• A run with failures is a SUCCESSFUL run — the failures are the whole value. A 9/12 run is the run that just told this team three things about their model. Lead with what did not pass; never soften it, never bury it under the passes.
• Tight loops. This skill is a tool, not a tutorial. One question per turn, at most two sentences of prose between phases.
• Don't paste SQL in chat. The script prints none, and neither do you. Same hard rule as agami-query.

Phase 0: Preflight

1. Plan-mode check per shared/plan-mode-check.md. This skill needs Bash + Read + Write — refuse if locked in plan mode. DO NOT write a plan file. DO NOT call ExitPlanMode. Refusal text: "I can't run an eval in plan mode — every case executes live SQL and the run writes a report. Switch to Auto or Edit Automatically mode (Shift+Tab to cycle) and re-invoke me with the dataset name."
2. Credentials present — read <artifacts_dir>/local/credentials for the active profile. If missing, invoke /agami-connect to set up first; this skill needs a working DB connection.
3. Model present — <artifacts_dir>/<profile>/datasource.yaml must exist. If not, invoke /agami-connect. This skill needs an introspected model: the run hands the generator the model's own tables and columns.
4. See what exists — run --list (Phase 1) before anything else. It reads no credentials and runs no query, so it is the cheap way to find out there is nothing to run.

Phase 1: Choose the dataset

python3 "$AGAMI_PLUGIN_ROOT/scripts/run_golden_eval.py" --profile <profile> --list

The JSON names each dataset with its total, confirmed and unconfirmed counts, plus datasets_dir — the directory a dataset would live in — and any findings the reader raised while reading them. Three outcomes:

• No datasets. Say so in one line, name the path (datasets_dir from the payload, i.e. <artifacts_dir>/<profile>/golden_datasets/<name>.yaml), and stop. Authoring one is not this skill's job — point at shared/golden-dataset-shape.md, which is the authority on every field. Carry its hard rule as you say so: never read another profile to learn the shape. A golden dataset is the business definitions and the answer key in one file, so globbing for a sibling's returns another tenant's questions together with the SQL that answers them. The reference has every field; a sibling profile is never the reference.
• Exactly one. Use it. No question — a bare invocation needs no argument, and the script picks it too.
• Several. Ask with AskUserQuestion, one option per dataset, each labeled with its confirmed / unconfirmed counts (orders — 14 confirmed, 2 unconfirmed) so the choice is informed by what can actually gate. If the user already named one in $ARGUMENTS, skip the question.

A dataset with zero confirmed items gets one line before you run it: "<name> has no confirmed cases, so this run reports scores but its verdict rests on nothing. See shared/golden-dataset-shape.md for confirming an answer key." Run it anyway if they want — a report with no gate is still a report.

If findings is non-empty, keep it: those are dataset-level breakages and they belong in Phase 3c.

Phase 2: Run

python3 "$AGAMI_PLUGIN_ROOT/scripts/run_golden_eval.py" \
  --profile <profile> --dataset <dataset> \
  > /tmp/agami-eval-<ts>.json

Then Read the file. The payload goes to a file rather than straight into the transcript because a run of forty cases is a large object and you need to look at parts of it more than once.

A run costs one model call plus one query per case, so tell the user what they are waiting for before it starts: "Running <dataset> — <N> cases, roughly <N> × 10–30s."

A run probes the generator once before the loop. One trivial question, and if the client cannot be started the run stops with exit 2 and says so — rather than spawning it once per case, failing identically each time, and presenting a wall of errors that reads like a catastrophic model regression. --skip-preflight refuses the probe, which is worth it only for a caller who already knows the client works and is counting tokens.

--timeout-s bounds a single generation (default 120). Raise it only if items are coming back with "the generator did not answer within the time this run allows".

Two flags narrow the run, and naming both is refused because they are two ways of selecting from the same dataset. --tag <name> is repeatable and runs the cases carrying any of the tags given, matched exactly — Smoke and smoke are different tags. --rerun-failures runs only what this dataset's last run recorded under failures. Both are for the non-interactive path; reach for them here only if the user asks for a slice or a re-run by name.

The exit code is the contract, and you should read it before the payload. 0 every confirmed case passed · 1 a confirmed case failed · 2 no verdict could be produced. A 2 is never a model regression — report it as a broken harness and do not open the failures table.

The script emits items already ordered — failures, errors, unscored, unconfirmed, passes — and summary.sections counts the rows in each. Render them in the order received and use those counts; do not re-sort, re-group or recount.

Phase 3: Present

3a — Summary line first

Ran <dataset> on <profile>: <sections.failure> failed, <sections.error> errored, <sections.unscored> unscored, <sections.unconfirmed> unconfirmed, <sections.pass> passed — run completed: <yes | no>.

Each placeholder names its own key under summary.sections; completed is summary.completed. The keys are spelled out because the payload also carries top-level summary.failed and summary.errored and they are different numbers — an unconfirmed miss counts in failed and is rendered under 3e, so a run can report failed: 2 beside one failure row. A verdict is not "zero failures". gating_failures counts items that were scored, so a run in which every generation errored has zero of them and is not green — and a run that stopped partway reports only the cases it reached. Read completed, gating_failures and errored together, and if completed is false say so on this line and again below: "The run stopped partway — <N> cases were never attempted, so this is not a clean result."

3b — Failures (lead with what did not pass)

### Failures

| Item | What went wrong | Gate |
|---|---|---|
| orders-paid-by-channel-2024 | 3 of 5 rows matched the answer key (accuracy 0.60) | — |
| orders-refunded-count | Right rows, but the statement never filtered on `status` | must_filter: status |

Item is item_key. What went wrong is reason, with accuracy and the two row counts where they add something. Gate is the entry from gates that fired, or — when the item simply disagreed with its answer key.

Under the table, one line of interpretation per failure — the same job the receipt does in reconcile:

orders-refunded-count — the rows are right and the filter is missing, so the model reached the answer another way. That is the kind of failure that passes today and breaks on next quarter's data.

3c — Errors and dataset findings

Two different things, and they must not share a table.

### Errors — the run could not produce an answer

| Item | What went wrong |
|---|---|
| products-count | The generator's answer did not carry a statement this run could read |

### Dataset findings — the file, not the answer

| Locator | Finding |
|---|---|
| orders.yaml[orders-last-quarter] | A relative question over a frozen answer key: the window slides, the SQL does not. |

A finding is not a scored failure. A rotted or unreadable case is broken, not wrong — nothing was judged, and counting it as a miss would blame the model for the dataset. The locator is <stem>.yaml[<case id>], which is what the author edits.

3d — Unscored

### Unscored — nothing could be compared

| Item | Why |
|---|---|
| orders-in-review | Both result sets came back empty, so there was nothing to compare |

Not a pass. An item is unscored when the comparator had nothing to work with — most often both sides returning no rows, typically because the question's window has outrun the data. The answer was never checked, so say that rather than letting a reader count it as agreement.

3e — Unconfirmed (kept visibly apart)

### Unconfirmed — reported, but they cannot gate

| Item | Score |
|---|---|
| payments-count | did not match the answer key (accuracy 0.00) |

These ran and they can never gate a run — nobody has confirmed their answer key, so failing a run on one would be gating on an unreviewed answer. Keep them under their own heading, after the failures, so a reader scanning for what broke never picks one up.

Most of them were also scored, but not all: an unconfirmed item can come back unscored like any other (both sides empty, say), and then accuracy is null rather than a number. Read status before you render a score — write "nothing could be compared" rather than the (accuracy 0.00) the template shows.

3f — Passes, then the drill-down

### Passed

12 cases reproduced their answer key: orders-count, customers-active, revenue-by-month, …

One compact line — the passes build the case, the failures drive the conversation. Then:

Report (both statements per case, side by side): <report>
Raw run: <artifact>
Re-run one dataset after a fix, or open the report to see what the model wrote.

<report> is report from the payload — the .html file the run just rendered, and the thing to hand over: it opens in a browser, needs no network, shows every case's question, its verdict, the table-set delta and the two statements side by side, and carries no result rows. <artifact> is artifact from the payload, the same run as JSON for anyone who wants to grep it.

Both files are where the answer key and the generated statement live. Handing over a path is the drill-down; reading either file aloud is not. If report is empty the render did not land — say so and point at the JSON, exactly as you would the other way round.

End the turn.

What a pass rate is, and is not

A run measures regression, not accuracy. Say so if anyone asks "how accurate is agami?" and reaches for this number.

Two reasons, and both are structural rather than fixable here. The generator in a run is handed the question and the model's context in one prompt, with every tool switched off and one attempt to answer — that isolation is what stops it reading the answer key, and it is the whole reason a score means anything. An agent answering the same question for real has get_datasource_schema, get_prompt_examples, execute_sql and list_datasources, and can look something up, try a statement and refine it. So a run scores a strictly weaker generator than the one that ships, and an item can fail here that production would get right on its second look.

And a golden dataset is not a random sample. It is the questions somebody would be embarrassed to get wrong, which is exactly what makes it useful and exactly what makes its pass rate unrepresentative. "12 of 15 passed" means those twelve still work. It does not mean 80%.

Hard rules

1. Never paste SQL in chat. The script prints no statement, and the artifact is a path you hand over, not a source to quote — do not read it out of the artifact to show the user what the model wrote. They open the file.
2. Never write or edit a dataset. Not to fix a rotted case, not to confirm an answer key, not to add the question a failure suggests. This skill reads and scores. Authoring is a separate job with its own reference (shared/golden-dataset-shape.md); route the user there.
3. Never re-judge a verdict. The score is deterministic and it came from agami-core. Report it, interpret it, and do not argue with it — no "this is arguably right", no re-reading rows to overturn a failure. If the score looks wrong, the case or the model is what changes, not the report.
4. A run with failures is a completed run, not an error. Report it as a result. Conversely a run of nothing but errors is not green: a verdict reads completed, gating_failures and errored together, never gating_failures alone.

Error handling cheat sheet