Accelint Eval Architect

Eval Architect

Decides whether and how to add automated evaluation to another skill, recommends the right framework (or honestly recommends none), and scaffolds a maintainable walking-skeleton harness. Defaults to cheap deterministic checks; gates LLM-judge calls behind explicit opt-in.

NEVER Do When Adding Evals

• NEVER reach for an LLM judge before ruling out deterministic verification — judge calls cost money per run, drift across model versions, and introduce false positives. A parser/compiler/schema/test-run check is cheaper, stable, and exact. The decision order in references/framework-matrix.md makes a judge unreachable until determinism is proven insufficient.
• NEVER hand-author golden/expected artifacts — they silently rot when the target's schema changes. (Real bug: ac-to-playwright's PERFECT-AC.plan.json used stale field names and failed a metric for months.) Generate goldens from the live schema/validator at scaffold time.
• NEVER pick thresholds blind — a threshold chosen without a baseline run manufactures a meaningless green checkmark. Scaffold thresholds as record-only, measure a baseline, then set numbers from the observed distribution. See references/calibration.md.
• NEVER scaffold the maximalist metric suite on day one — an unmaintained comprehensive eval is worth less than a maintained walking skeleton. Ship one fixture, one metric, one passing test, one regression test; document the extension path. Exception: detector/review skills ship recall AND false-positive-resistance as an inseparable pair — either alone rewards degenerate behavior (flag everything, or flag nothing).
• NEVER ship a metric without a regression test that proves it can fail — a metric that always passes is decoration. Every metric gets a planted-broken input that drives it below threshold.
• NEVER force a persona×scenario grid onto a single-mode skill — when there is one user and one mode, flat fixtures across an input-quality gradient are clearer. Derive the taxonomy from the skill; don't impose it. See references/test-design.md.
• NEVER leave eval source untracked — results/, .venv/, __pycache__/ are gitignored, so it is easy to orphan the source files alongside them (this happened to the reference impl and required bytecode recovery). Commit eval source before the first run.
• NEVER recommend a Python judge framework for a Node-only skill unless judgment genuinely requires it — house toolchain fit lowers the maintenance barrier. Match the harness to the target's existing package.json/pyproject.toml.
• NEVER evaluate taste-based output with an LLM judge — for visual, creative, or aesthetic output, LLM judges are weaker and less honest than humans. Recommend a structured human-review checklist instead. See references/frameworks/human-review.md.

Before Recommending an Eval, Ask

Verifiability (decides judge vs deterministic vs none)

• Can a parser, compiler, schema, or test-run check this output with no judgment? If yes → deterministic-only, no judge.
• Is any part of "good" inherently subjective or taste-based? That part is human-review, not automated — be honest about the ceiling.
• Is the output partially verifiable (structure deterministic, quality judgmental)? → hybrid: deterministic gates plus a thin judge layer for the judgment-only slice.

Cost vs value

• Is this skill used enough that regressions matter? Low-traffic + low-stakes output may not warrant any eval. Saying "don't build one" is a valid, valuable outcome.
• Does the judgment slice justify per-run judge cost? Estimate tokens before recommending a judge framework.

Toolchain fit

• Is the target a Node or Python package? Match the deterministic harness to it (vitest vs pytest).
• Does a DeepEval harness already exist in the repo to reuse? Its reporter, judge adapter, and noise filters transfer — house consistency for judge layers.

Maintenance

• Who recalibrates thresholds after a model upgrade? If nobody, scope the eval smaller.
• Will the goldens track the schema automatically, or rot? Generate, don't hand-write.

How to Use

This skill uses progressive disclosure. Detect the mode first, then load only the references each step names.

Modes

The routing rules (which phrases and repo states select which mode) live in Mode 0 below.

• ASSESS — read a target, classify it, recommend a framework (or none). Read-only.
• SCAFFOLD — build the walking-skeleton eval for the approved framework, after ASSESS.
• AUDIT — run the decay checklist against a target that already has an evals/ dir.
• EXTEND — add metrics/fixtures to an existing eval.

Reference map (load on demand)

• Reading + classifying a target skill → references/assessment.md
• Reading + classifying a target tool repo / pipeline → references/tool-repo-assessment.md
• Choosing a framework → references/framework-matrix.md
• Designing the test matrix + metrics → references/test-design.md
• Decomposing a pipeline into stages + per-stage metrics → references/pipeline-decomposition.md
• Building/curating a RAG gold set → references/gold-set.md
• Setting thresholds from a baseline → references/calibration.md
• Auditing an existing eval → references/audit.md
• Framework specifics → references/frameworks/deepeval.md, ragas.md, deterministic-vitest.md, deterministic-pytest.md, human-review.md

Do NOT load all references at once. Each step below names what it needs.

Workflow

Mode 0 — Detect target type, then mode

Target type (decides which assessment path):

path has SKILL.md at root                         → target_type = skill
path is a repo, no SKILL.md, has a package manifest → target_type = tool-repo
ambiguous                                          → ASK; never guess

Mode:

target has no evals/ + "add eval" intent    → ASSESS, then offer SCAFFOLD
"scaffold it" / approval after an ASSESS    → SCAFFOLD
"audit" / "is my eval good" / evals/ exists → AUDIT
"add a metric" / "extend" + evals/ exists   → EXTEND
"should I eval X" / "which framework"       → ASSESS only

ASSESS

1. Read the target.
   • target_type = skill → load references/assessment.md: read SKILL.md, references/, scripts/, assets/, toolchain.
   • target_type = tool-repo → load references/tool-repo-assessment.md: read README + manifest + entry points, then interview the developer for pipeline stages, how each is invoked, the eval boundary, and whether a gold set exists. You cannot reverse-engineer a pipeline from files alone — confirm stages + invocation with the developer.
   • If anything required is unreadable or unconfirmed, ask — never invent facts about a target you did not read.
2. Produce the eval profile (skill schema in assessment.md; tool-repo schema in tool-repo-assessment.md). Populate unread_or_uncertain; resolve before recommending.
3. Run the verifiability gate (in framework-matrix.md). For a pipeline, run it per stage (see references/pipeline-decomposition.md): ingestion + retrieval are usually deterministic; only answer-generation is a judge slice.
4. Recommend a framework. Load references/framework-matrix.md. State the recommendation, the one-sentence reason it beats the runner-up, and a token-cost estimate. For RAG, recommend the answer-layer framework (Ragas vs DeepEval) per case, not by default.
5. Design the test matrix. Load references/test-design.md (skill) or references/pipeline-decomposition.md (pipeline). Propose metrics grouped by dimension/stage, each with a regression-test sketch. For RAG, flag that the gold set is the prerequisite (references/gold-set.md).
6. Present the recommendation summary (format in framework-matrix.md). Stop. Wait for approval, redirect, or decline.

SCAFFOLD (only on approval)

7. Copy the framework starter from assets/templates/<framework>/ into the target's evals/. Use scripts/scaffold_eval.py for the mechanical copy + placeholder substitution. For a RAG pipeline the starter is assets/templates/rag/ (deterministic ingest + retrieve metrics PLUS a gated answer-layer metric).
8. Generate the walking skeleton: the minimum that proves the pipeline can be measured — for a skill, one fixture + one metric + pass + regression; for RAG, a small curated gold set + section_coverage (ingest) + recall@k (retrieve) with regression tests + one gated answer metric. Nothing more.
9. Generate goldens from the live schema/validator if one exists — never hand-typed. For RAG there is no schema; bootstrap the gold set with scripts/bootstrap_goldset.py, then require human curation (references/gold-set.md) — an un-curated gold set graded by an auto-generated judge is circular.
10. Set all judge thresholds to record-only; wire the runner into the target's build (npm script or pyproject); write README.md + DESIGN.md stubs (DESIGN.md ships with a pre-filled "Known follow-ups").
11. Run the don't-lose-it check: confirm source is git-tracked, results//.venv//__pycache__/ ignored, and tell the developer to commit before the first run.
12. Hand off with calibration instructions (references/calibration.md): run N baselines, then set thresholds from the distribution.

AUDIT

13. Load references/audit.md. Inventory the existing harness, run the decay checklist (stale goldens vs schema, metrics that can't fail, sentinel thresholds, renamed-fixture drift, scenarios missing metrics, judge rubrics penalizing correct behavior, untracked source). Emit findings ranked by severity. Do not auto-fix without approval.

EXTEND

14. Design the new metric per references/test-design.md (or references/pipeline-decomposition.md for pipelines). The regression-test rule applies to every added metric, same as at scaffold time.
15. To pull in another template's files (e.g. adding a judge layer to a deterministic harness), use scripts/scaffold_eval.py --layer — it copies only files that don't already exist and reports collisions to merge manually (typically conftest fixtures). --dest evals-judge places a fully separated layer instead.
16. Re-run the existing suite green first — extending a broken eval buries the regression you're about to care about.
17. Recalibrate if the extension adds judge metrics (references/calibration.md); new judge metrics start record-only like always.

Important Notes

• Two LLM roles. Every judge-based eval has an SUT (the model being graded) and a Judge (the model grading it) — often the same alias, conceptually distinct. Keep them separate in every recommendation and every scaffolded harness; conflating them is the #1 source of developer confusion.
• Verifiability before everything. The single most common mistake is using a judge where a parser would do. The matrix enforces deterministic-first; honor it.
• The honest "no" is a feature. "Don't build an automated eval — here's the human-review checklist" and "deterministic-only, skip the judge" are first-class outcomes, not failures of the skill.
• Tool-repo comprehension is interactive, not magic. The skill cannot reverse-engineer an arbitrary pipeline; it reads what it can and interviews the developer for stages + invocation. Refuse to recommend until both are confirmed.
• RAG gold sets must be human-curated. Bootstrap drafts are a starting point, never trusted output — and never let the gold set and the judge come from the same unreviewed pass.
• Template version floors drift too. Templates use floors (deepeval>=4, ragas>=0.2, litellm>=1.55), not hard pins — but a floor goes stale when a framework ships a breaking major. Sanity-check against the target's environment when scaffolding; tell the developer to commit the lockfile for reproducibility.
• Scope: scaffolds DeepEval, deterministic-vitest, deterministic-pytest, human-review-checklist, and rag (the deterministic ingest+retrieve layer AND the gated Ragas answer layer in one template — there is no separate "deterministic-retrieval" scaffold key). Promptfoo and Inspect AI are recommended when they fit but handed off. Tool-repo support targets RAG / retrieval pipelines; other tool types are recommended-and-handed-off, not yet scaffolded.