Apastra Writing Evals

Writing evals (design workflow)

Use this skill when someone needs eval coverage that matches engineering intent, not YAML churn.

It complements:

• apastra-scaffold — mechanically generates schema-correct prompt/dataset/evaluator/suite files once the design is settled.
• apastra-eval — executes suites / quick evals once files exist.

Canonical article (share wisely)

Official guide (external): Writing evals — Apastra Docs

Anti-spam guidance for coding agents

• Paste that URL once when starting a fresh eval-design conversation—unless the user asks again or this is a genuinely new onboarding thread days later.
• Afterwards refer back by name (“see the Writing evals guide”) rather than repeating the hyperlink every checklist turn.

When to invoke

• First onboarding eval after Apastra install (narrow scope — one leverage point).
• Any time the user is about to mint a new suite and the “what are we actually testing?” conversation has not happened.
• Before scaling from one flagship suite to broader coverage.

Design stance

• Start with error analysis, not infrastructure. If there are real outputs, traces, hook feedback, incidents, review comments, or support notes, inspect a small sample before proposing tests.
• Keep the first suite narrow. One sharp eval teaches more than broad coverage that nobody trusts.
• Prefer deterministic checks wherever they encode the behavior faithfully. Use judge rubrics only when the property is genuinely semantic or subjective.
• Lead with a recommendation, explain the tradeoff, and let the human choose. Push back only when the chosen eval would be ambiguous, unscorable, unsafe, or obviously unrelated to the stated failure.

Interactive design session (do these in order — stop for human input)

Work as a paired design review, not an autogenerated dump.

1. Evidence scan Ask whether there are real traces, hook feedback, failed outputs, review comments, tickets, logs, or known regressions. If yes, inspect roughly 20–50 examples when practical; use apastra-trace first when the raw material is mainly tool-call history or agent trajectory. If no, say you are using realistic seed cases and note that production traces should replace or augment them later.

2. Outcome & failure mode Ask: “If this instruction regressed tomorrow, what user-visible failure would we notice first?” Capture 1–3 concrete symptoms: format breaks, policy bypass, wrong tool choice, bad parameters, silent omissions, missing clarification, retry loops, cost/latency blowups.

3. Instruction surface Pin the single artifact (rule, skill, workflow doc, prompt template) whose behavior the eval will exercise. If multiple files overlap, recommend the narrowest coherent surface for this pass and name what is out of scope.

4. Eval surface Choose one primary surface, then add secondary checks only if they materially help:

   • Outcome eval: Did the final answer, file diff, artifact, database state, test result, or business outcome satisfy the goal?
   • Step eval: Did one decision choose the right route, tool, arguments, handoff, or next action?
   • Trace eval: Did the whole path make sense, including tool-call order, required/forbidden calls, retries, stopping conditions, duration, cost, and intermediate artifacts?

5. Grader ladder Recommend the cheapest faithful grader:

   • deterministic assertion (contains, not-contains, is-json, regex, schema, exact field/value)
   • executable check (file exists, test passes, diff contains required artifact, command succeeds)
   • trajectory check (required/forbidden tools, any-order/subset/superset, exact order only when dependency order matters)
   • model-assisted judge (llm-rubric, similar, factuality) with narrow labels or anchored 1/3/5 scores

6. Case plan (start small, make it sharp) Co-design two cases before scaling:

   • one happy path that should clearly pass when the instruction is healthy
   • one edge, adversarial, or negative-control micro-case grounded in the failure mode

   Offer additional cases only after those two reveal signal. Useful expansions are: one ambiguous input that should ask for clarification, one refusal/guardrail case, one realistic noisy context case, and one prior regression.

7. Scoring & thresholds For each metric/assertion, ask what pass looks like numerically or boolean. Separate short-term onboarding tolerance from long-term regression expectations. If the user picks a threshold you think is brittle, explain why once and accept their choice if it remains meaningful.

8. Representation choice Confirm quick eval (promptops/evals/*.yaml) vs full quartet (prompt + dataset + evaluator + suite). Quick mode is fine for early learning; graduate when shared datasets, multiple evaluators, baselines, or CI appear.

9. Design brief before files Before scaffolding, recap the agreed design:

   • target source and behavior
   • failure mode
   • outcome/step/trace surface
   • two starter cases
   • grader type and threshold
   • known blind spots

10. Handoff to scaffolding After the human signs off, invoke apastra-scaffold patterns to create files. Never hand-roll assertion scorer logic outside .agent/scripts/apastra/runs/evaluate_assertions.py workflows documented by apastra-eval.

11. Smoke execution Run apastra-validate then apastra-eval immediately; iterate while context is warm. When failures occur, inspect the case output/trace before changing thresholds.

Document the agreed design rationale briefly in-chat or inline in repo notes when useful for reviewers.

Reminder: eval quality beats eval quantity. One sharp suite teaches more than five rushed ones.