Ramp And Verify

Ramp and Verify

Every successful journey through this library ends the same way: a candidate won on a frozen eval and a route decision says ship it. This worker owns the last mile — taking that candidate from 0% to 100% of live traffic on the gateway dial without trusting a single lab number, and producing the measured before/after that makes any savings statement honest. It starts where ../compare-model-sweep/SKILL.md and ../use-understudy-gateway/SKILL.md stop.

Resolve CLI

Prefer the installed understudy binary. If it is unavailable inside a repo checkout, run through the package script:

npm run build
node dist/bin.js status --json

Safety Gates

• Every traffic change is an explicit, approved action. Name the workload, the model id, the old and new percentage, and get approval in the current thread before each routes set. Never ramp two tiers in one step.
• Disclose capture-on-by-default. Setting a model route enables request capture for that workload unless capture is explicitly disabled — tell the developer this before the first route write and confirm the capture setting they want. The hosted capture behavior is documented at docs.understudylabs.com/concepts/capture.
• Snapshot before every change. The CLI writes a route snapshot before route writes; verify it exists so routes rollback has something to restore.
• No savings claim without a claim packet. Measured routed-vs-passthrough deltas go into the evidence contract from ../optimize-workload/SKILL.md (claim.json); a tier that "looks cheaper" is not a claim.
• Captures contain raw request/response bodies. Read them through the CLI's redacted, metadata-first views; full payload export is opt-in, file-only, and approval-gated.

Pre-ramp gates (all must pass)

1. Frozen-eval verdict exists — a sweep/optimization result on frozen splits with the incumbent baseline recorded, or a pass launch verdict from ../simulate-before-launch/SKILL.md for the specific change being ramped (its contract axes — schema validity, tool-call validity — are the same ones step 3 verifies from captures).
2. Repeat-replay stability. Re-run the candidate on the frozen rows in incremental batches. Three repeats may serve as a plumbing smoke, but size N for the agreed instability tolerance and increase it when results sit near the threshold or vary across batches. Bucket each row: all-repeats-match / some / none (the procedure lives in ../compare-trajectories/SKILL.md). Rows that never repeat are stochastic pockets — assign each a disposition: fallback (route to incumbent), shadow (observe, don't serve), or requires-fresh-traffic. Stop if >~5% of rows are unstable.
3. Fallback exists. The non-routed remainder passes through to the incumbent by construction; confirm the workload's passthrough path works (gateway probe) before sending anything to the candidate.

Desktop app export packets under ~/.understudy/exports/ are admissible for gate 1 once verified — shape, sibling-JSONL hash, split identity, and cost basis all check out. The admission checklist and verification transcript live in reference.md.

The ladder

Default tiers: 5% → 25% → 100% (note: routes set defaults --traffic-pct to 10 when not specified, so pass --traffic-pct 5 explicitly for the first tier). The dial's hosted split semantics are documented at docs.understudylabs.com/concepts/routing. At each tier:

1. understudy routes set <workload> --project <p> --model-id <id> --traffic-pct <pct> — after approval.

2. Soak. Hold the tier for an agreed window (hours at 5%, a day at 25%) while traffic accumulates.

3. Verify from reporting + captures. Check the reporting endpoints first for a fast aggregate view:

   GET .../provider-health?window=<soak-window>
   GET .../status?window=<soak-window>
   

   (docs) Use ../check-routing-health/SKILL.md for the full diagnostic flow. Then pull the window's captures (metadata-first) and compare routed vs passthrough on the same period: error/status-code rate, latency distribution, schema/parse validity of outputs, and token/cost per call. The routed cohort must hold the lab quality bar on whatever the workload's validator can score from captures.

4. Check determinism on a decision-sized stratified set. Re-run routed rows against the candidate, expanding the set when variance, rare strata, or failures remain uncertain. If repeats disagree materially more than the pre-ramp measurement, stop the ramp and re-diagnose. Follow ../capture-evidence/references/evaluation-evidence-gates.md.

5. Advance or roll back. Advance only when the tier window is clean. Rollback triggers — any one fires understudy routes rollback (or routes clear) immediately: error rate or schema-validity regression vs passthrough, tail-latency regression beyond the agreed bound, instability above the pre-ramp measurement, or the developer's call. After a rollback, do not re-ramp until the root cause is fixed and the pre-ramp gates pass again on a fresh eval.

After 100%

Keep the incumbent reachable (snapshot retained, rollback tested) for an agreed observation window. Then assemble the before/after: incumbent baseline vs routed steady-state on quality-from-captures, latency, and cost/call — into the claim packet. That packet, not the ramp log, is what any cost or quality statement cites. It inherits the claim-packet rigor fields from ../optimize-workload/SKILL.md: the trivial-agent floor (the claim is invalid if a do-nothing agent also clears the quality bar) and bootstrap confidence intervals on the before/after quality delta — an overlap means "no measured regression", never "measured improvement" (see ../../docs/benchmark-rigor.md).

Output Standard

End with: the workload/project/model ids and the tier history (pct, window, verdict per tier); the pre-ramp gate results (stability fraction, disposition counts); per-tier routed-vs-passthrough table (errors, latency, validity, cost/call); any rollbacks and their trigger; the current route state and snapshot path; result type (live); and the claim-packet status with the one recommended next action.

References

• ../use-understudy-gateway/SKILL.md — route/capture command surface and auth readiness.
• ../compare-trajectories/SKILL.md — repeat-replay stability procedure and dispositions.
• ../compare-model-sweep/SKILL.md — the frozen-eval verdict required by the pre-ramp gate.
• ../simulate-before-launch/SKILL.md — the offline launch verdict for a single proposed change (the other producer gate 1 accepts).
• ../optimize-workload/SKILL.md — the claim-packet contract for any measured savings statement.
• ../check-routing-health/SKILL.md — read-only diagnostics from the reporting endpoints (routing-status, provider-health, compact status).
• Hosted contracts on the docs site — routing, capture, reporting & health, and the control-plane API behind routes set/rollback/clear and captures list.