# Apastra Writing Evals

> Interactive PromptOps eval design — what behavior to pin down, cases, scoring, thresholds, and outcome/step/trace evidence. Deep link to Writing evals once per session; pair with scaffold + apastra-eval.

- Skill: `bintzgavin/apastra-writing-evals` (Agent Skill)
- Install (CLI): `npx skillmds@latest add bintzgavin/apastra-writing-evals`
- Raw SKILL.md: https://api.skillmd.com/api/skills/bintzgavin/apastra-writing-evals/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: bintzgavin (https://skillmd.com/u/bintzgavin)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/bintzgavin/apastra-writing-evals

---


# Writing evals (design workflow)

Use this skill when someone needs **eval coverage that matches engineering intent**, not YAML churn.

It complements:

- **`apastra-scaffold`** — mechanically generates schema-correct prompt/dataset/evaluator/suite files once the **design** is settled.
- **`apastra-eval`** — executes suites / quick evals once files exist.

## Canonical article (share wisely)

Official guide (external): [Writing evals — Apastra Docs](https://bintzgavin-apastra-14.mintlify.app/guides/writing-evals)

### Anti-spam guidance for coding agents

- Paste that URL **once** when starting a fresh eval-design conversation—unless the user asks again or this is a genuinely new onboarding thread days later.
- Afterwards refer back by name (“see the Writing evals guide”) rather than repeating the hyperlink every checklist turn.

## When to invoke

- **First onboarding eval** after Apastra install (narrow scope — one leverage point).
- Any time the user is about to mint a new suite and the “what are we actually testing?” conversation has not happened.
- Before scaling from **one** flagship suite to broader coverage.

## Design stance

- Start with **error analysis**, not infrastructure. If there are real outputs, traces, hook feedback, incidents, review comments, or support notes, inspect a small sample before proposing tests.
- Keep the first suite narrow. One sharp eval teaches more than broad coverage that nobody trusts.
- Prefer deterministic checks wherever they encode the behavior faithfully. Use judge rubrics only when the property is genuinely semantic or subjective.
- Lead with a recommendation, explain the tradeoff, and let the human choose. Push back only when the chosen eval would be ambiguous, unscorable, unsafe, or obviously unrelated to the stated failure.

## Interactive design session (do these in order — stop for human input)

Work as a **paired design review**, not an autogenerated dump.

1. **Evidence scan**
   Ask whether there are real traces, hook feedback, failed outputs, review comments, tickets, logs, or known regressions. If yes, inspect roughly 20–50 examples when practical; use **`apastra-trace`** first when the raw material is mainly tool-call history or agent trajectory. If no, say you are using realistic seed cases and note that production traces should replace or augment them later.

2. **Outcome & failure mode**
   Ask: “If this instruction regressed tomorrow, what **user-visible** failure would we notice first?” Capture 1–3 concrete symptoms: format breaks, policy bypass, wrong tool choice, bad parameters, silent omissions, missing clarification, retry loops, cost/latency blowups.

3. **Instruction surface**
   Pin the **single** artifact (rule, skill, workflow doc, prompt template) whose behavior the eval will exercise. If multiple files overlap, recommend the **narrowest** coherent surface for this pass and name what is out of scope.

4. **Eval surface**
   Choose one primary surface, then add secondary checks only if they materially help:
   - **Outcome eval:** Did the final answer, file diff, artifact, database state, test result, or business outcome satisfy the goal?
   - **Step eval:** Did one decision choose the right route, tool, arguments, handoff, or next action?
   - **Trace eval:** Did the whole path make sense, including tool-call order, required/forbidden calls, retries, stopping conditions, duration, cost, and intermediate artifacts?

5. **Grader ladder**
   Recommend the cheapest faithful grader:
   - deterministic assertion (`contains`, `not-contains`, `is-json`, `regex`, schema, exact field/value)
   - executable check (file exists, test passes, diff contains required artifact, command succeeds)
   - trajectory check (required/forbidden tools, any-order/subset/superset, exact order only when dependency order matters)
   - model-assisted judge (`llm-rubric`, `similar`, factuality) with narrow labels or anchored 1/3/5 scores

6. **Case plan (start small, make it sharp)**
   Co-design **two cases** before scaling:
   - one **happy path** that should clearly pass when the instruction is healthy
   - one **edge, adversarial, or negative-control micro-case** grounded in the failure mode

   Offer additional cases only after those two reveal signal. Useful expansions are: one ambiguous input that should ask for clarification, one refusal/guardrail case, one realistic noisy context case, and one prior regression.

7. **Scoring & thresholds**
   For each metric/assertion, ask what **pass** looks like numerically or boolean. Separate short-term onboarding tolerance from long-term regression expectations. If the user picks a threshold you think is brittle, explain why once and accept their choice if it remains meaningful.

8. **Representation choice**
   Confirm **quick eval** (`promptops/evals/*.yaml`) vs **full quartet** (prompt + dataset + evaluator + suite). Quick mode is fine for early learning; graduate when shared datasets, multiple evaluators, baselines, or CI appear.

9. **Design brief before files**
   Before scaffolding, recap the agreed design:
   - target source and behavior
   - failure mode
   - outcome/step/trace surface
   - two starter cases
   - grader type and threshold
   - known blind spots

10. **Handoff to scaffolding**
    After the human signs off, invoke **`apastra-scaffold`** patterns to create files. Never hand-roll assertion scorer logic outside `.agent/scripts/apastra/runs/evaluate_assertions.py` workflows documented by **`apastra-eval`**.

11. **Smoke execution**
    Run **`apastra-validate`** then **`apastra-eval`** immediately; iterate while context is warm. When failures occur, inspect the case output/trace before changing thresholds.

Document the agreed **design rationale** briefly in-chat or inline in repo notes when useful for reviewers.

---

**Reminder:** eval quality beats eval quantity. One sharp suite teaches more than five rushed ones.