Optimize Agentic Workload

Optimize Agentic Workload

Use this worker when the workload is an agentic loop: an LLM that plans over multiple turns and calls tools (web search, retrieval, REST/SDK calls, code execution) before finishing. Freeze every non-target variable within each experiment; model/route comparisons hold the tools fixed, while a named implementation arm may isolate one deployable tool, app, or harness change. The goal is to pick the route and implementation you would ship on a multi-objective basis — quality and latency and cost (and, for workflows that write, side-effect safety). The route is backend-agnostic: use the existing harness, provider-native evaluation or training, the public CLI, or a small versioned app/harness change when the measured failure calls for it.

The one discriminator that changes the playbook is whether the loop mutates state:

• Read-only search loops — web/agentic search, retrieval, lookup tools. Success depends on how the agent searches and the final answer; determinism comes from snapshotting live tool outputs. Harness specifics: references/read-only-search.md.
• State-mutating API workflows — the agent discovers or selects endpoints, follows policy docs, performs writes across business systems, and is judged by final state plus policy compliance. Determinism comes from seeded, resettable state; safety (no forbidden writes) is a first-class objective. Harness specifics: references/state-mutating-workflows.md.

Everything else — the artifact contract, measured baseline, workload-specific decision contract, evidence-driven intervention choice, and stronger RL gates — is shared.

This is not the RL handoff skill. Reach for ../prepare-verifier-handoff/SKILL.md only when the evidence shows the residual requires reinforcement learning of stateful behavior and its reward and renderer gates pass. Supervised fine-tuning or distillation is a separate intervention and can be selected earlier when correction labels or deterministic verifier targets support it.

Safety Gates

Default to the path with the highest expected progress toward the multi-objective decision under hard constraints, not the cheapest rung. State the expected quality, latency, safety, time, and spend tradeoffs; follow ../understudy/reference.md → Outcome-first spend posture. Launching a named bounded run plan authorizes its declared provider calls, uploads, hosted jobs, tool calls, evaluation, receipts, and cleanup. Do not pause for phase-by-phase confirmation. Ask again only before expanding its displayed data, destination, spend, retention, credentials, or production-write scope.

Use seeded fixtures and resettable sandboxes when the decision requires repeatability. Treat production writes as a separate envelope from model and training writes. Follow the repo public boundary in ../../docs/privacy-and-data-boundaries.md for prompts, completions, tool outputs, traces, datasets, repo paths, and secrets. Do not print or commit sk_* values; let ../use-understudy-gateway/SKILL.md inject them into the child process only.

Resolve CLI When Using An Understudy Route

Prefer the installed understudy binary. If it is unavailable inside a checkout:

npm run build
node dist/bin.js status --json

When To Use

Use this skill when all of these hold:

• the workload runs a tool-calling loop, not a single prompt-in/answer-out call;
• success depends on how the agent acts (turn count, which tools, in what order, what it writes), not just the final string;
• the tool contract can be frozen for model comparisons, and any implementation change can be isolated as its own named arm;
• the developer wants to compare candidate policy models, or shrink a frontier model down to a cheaper one without losing quality.

If the workload emits one output with no tool loop, route to ../capture-evidence/SKILL.md then ../optimize-workload/SKILL.md instead — single-output optimization does not need a tool environment.

Flow

1. Confirm it is agentic and pick the lens. Inspect the workload for a multi-turn loop and tool calls. State the fixed tool set and policy model for route comparisons, plus any separately isolated implementation variable. Then classify: do any tools mutate state? Read-only → references/read-only-search.md; state-mutating → references/state-mutating-workflows.md. If it is single-output, hand back to capture-evidence.

2. Adopt a runnable harness. Read-only loops use a verifiers environment (vf.Environment + Rubric; the vf-eval command is one supported harness) or another replayable provider-native/application runner. State-mutating workflows use an existing benchmark/sandbox runner with a deterministic reset (seeded state, fixed API schemas, fixed policy docs, final-state validator). A small deployable app or harness change is allowed when it directly addresses a measured failure; version it and evaluate it as a distinct arm instead of silently changing the comparison. Either way, capture the runnable contract into the .understudy/capture-evidence/ artifact contract — each reference documents the env → artifact bridge for its shape.

   Before using model scores, pass the shared evaluation evidence gates in ../capture-evidence/references/evaluation-evidence-gates.md. In particular, run a synthetic read-then-write trajectory through the exact driver for every model family. The driver must append the read result, continue the loop, execute the terminal write, and score final state; an intermediate tool call is never a no-op verdict.

   If Desktop produced understudy.environment_proposal.v1 from a JSONL drop, treat status: executable as meaningful only after understudy training validate-environment-proposal --proposal <path> passes. An Understudy draft with status: needs_verifier is a proposal, not a harness or score; author the missing parser/environment/oracle/sentinels and rerun deterministic validation before any model comparison.

3. Define multi-objective success. Quality is a per-criterion LLM-judge or final-state rubric that returns natural-language why/what-to-change feedback, not a bare score. Latency and cost come from the rollout records (turn counts, tool/API call counts, timing, token usage) — read those instead of inventing a meter. State-mutating workflows add a side-effect-safety axis (forbidden writes, invalid requests, retries). Record the axes, workload-specific acceptable-regression/non-inferiority bands, and any hard constraints in metric.json. Contract requirements and safety requirements marked hard by the developer remain zero-tolerance; statistical or multi-objective tradeoffs cannot waive them.

4. Freeze determinism. Read-only: live tool calls are non-deterministic, so freeze the query set and snapshot/cache the tool outputs so the harness replays reproducibly and the holdout stays clean. State-mutating: record a deterministic reset — seeded state, schemas, policy docs, task rows, allowed endpoints, clock, seed, network boundary — so each run starts from a known state and emits a request log.

5. Run the incumbent baseline before optimizing. Execute the frozen harness on the sanctioned train/dev set, write per-task results, and bind harness_sha256, metric_sha256, and splits_sha256 into baseline.json. Confirm coverage across completed-execution strata, including rare or high-consequence tool paths, and inspect the actual trajectories behind at least one pass, each reported failure class, each surprising delta, and a counterexample to the proposed headline. Do not optimize until the baseline is measured and those checks pass. A smaller pilot may validate plumbing, but it is not the baseline evidence. Expand the cohort whenever estimates remain unstable, important strata are underfilled, or review discovers new failure classes.

6. Attribute the multi-turn gap before intervening. Read the rollouts, not just the final score, and tag where reward is lost: wrong tool/endpoint, wrong argument value, result-propagation (mis-copying a value a prior tool returned into a later call), failure to recover from a tool error, forbidden or missing writes, or non-termination. Single-turn / next-tool-call imitation scores are a leading indicator only — they cannot see result-propagation, recovery, or termination, which exist only inside the running environment. Let the attribution pick the highest-leverage rung likely to close the gap: output-contract repair (prefill / format / parser / schema), tool-access or endpoint-catalog repair, model A/B, prompt / GEPA (automatic prompt evolution), supervised fine-tuning/distillation, or RL. Rank the eligible rungs by expected objective gain, confidence, time, spend, and reversibility; start with the highest expected value rather than a fixed sequence. Use a cheaper rung first only when evidence says it can solve the attributed failure.

7. Candidate intervention — compare models and routes. When attribution points to model capability or provider/runtime behavior, compare deployable candidates through any backend that can honor the frozen contract. The Understudy CLI is one route:

   understudy models list --json
   understudy workloads route <workload-id> --project-id <project-id> \
     --model-id glm-5.1 --traffic-pct 100
   understudy run -- <harness command>
   

   List public model options, route the project workload to a chosen model, then run the frozen harness through the gateway with understudy run; a provider-native or existing application runner is equally valid when it preserves the same rows, prompt, tools, metric, seed/reset, and data boundary. Compare quality vs latency vs cost (vs side-effect safety) across candidates under the decision contract and pick the route you would ship. For keyless accounts, prefer a managed-catalog sweep on a cleared/no-route workload before traffic-split A/B. Prerequisite for a traffic split: the non-routed passthrough share needs a configured managed provider credential or BYO key so untouched traffic still completes. Clear a route with --clear. Routing detail lives in ../use-understudy-gateway/SKILL.md. For state-mutating workflows, A/B is often simpler: run the same harness rows twice with only the model changed (see the reference).

8. Candidate intervention — repair prompt, tools, contract, or harness. When the attributed gap is instructional or structural, test the smallest deployable correction: prompt/GEPA, output parser or schema, tool descriptions/retrieval, retry policy, or a versioned app/harness change. If a cheaper model wins on latency and cost but trails on quality, a train/dev-only GEPA pass against the feedback-rich rubric may close the gap while keeping the latency/cost win. When the workload already lives in a promoted benchmark dir with frozen splits, run the automatic loop directly:

   # terminal 1 — the only thing that executes models
   understudy runs execute --benchmark <benchmark-dir> --watch
   # terminal 2 — proposes, queues override arms, waits, scores
   understudy benchmarks evolve <benchmark-dir> --model <candidate-id> --budget-runs 6
   

   evolve is GEPA-style prompt evolution over prompt_overrides run arms: an authoring model proposes system-prompt suffixes from the failure evidence (per-class tool-call rejection counts and unmet contract obligations read from the run journals), each proposal runs as a labeled override arm, and every generation is recorded in evolution.jsonl (suffix + sha256 + scores). See ../../docs/prompt-evolution.md for the loop and budget guidance. Claim rules mirror ../optimize-workload/SKILL.md: it evolves on train, selects the champion on dev, touches the sealed holdout exactly once for the final champion-vs-bare run, and reports quality improvement only when that holdout run's paired 95% CI clears the prespecified superiority threshold. A route may still be selected after demonstrated quality non-inferiority or by a prespecified multi-objective rule, but a no_win/ unverified/inconclusive superiority verdict must never be presented as an improvement. For single-output workloads, hand off to ../optimize-workload/SKILL.md instead; never tune on holdout.

   Prompt experiments as run arms (prompt_overrides). When the workload lives in a trace-compiled benchmark directory, test a prompt candidate without touching the app: queue a run whose request carries prompt_overrides — each {arm_label, model, system_prompt_suffix} arm appends the suffix file's text to every task's system prompt at rollout time (understudy runs queue --prompt-override <arm_label>=<model>=<suffix-file>). The canonical use is the SOP-suffix experiment: distill the workload's standard operating procedure (order of operations, argument conventions, termination rules) into a short suffix and run it as its own arm beside the bare model on the same frozen tasks — one run, same scorer, per-arm rows. Operating detail in ../operate-benchmark-lab/SKILL.md.

9. Candidate intervention — supervised fine-tuning or distillation. Select this early when the failure evidence supplies learnable correction pairs, teacher trajectories, or deterministic verifier labels and a trained candidate can be evaluated in the same end-to-end harness. Do not require model A/B or prompt optimization to fail first. Freeze the dataset lineage, keep holdout sealed, declare provider/data/retention/spend bounds, and compare the trained route against the incumbent under the same hard safety and contract constraints. Treat next-action imitation as diagnostic unless the end-to-end rollout also validates result propagation, recovery, termination, and final state.

10. Escalate to RL only as a true handoff, behind three gates. When the residual is genuinely stateful multi-step behavior, route to ../prepare-verifier-handoff/SKILL.md. First confirm: (a) the attribution in step 6 shows cross-turn reasoning is the residual, not format/argument-value (which supervised or deterministic repairs can address); (b) the reward is dense, not strict — a binary/strict reward can be constant within a group, giving zero advantage and no gradient (paid-for, wasted steps); and (c) the model has a first-class multi-turn GRPO trainer and renderer (e.g. NVIDIA Nemotron-3 does; Google Gemma-4 does not yet), or the RL run is wasted before it starts. This repo never runs that training.

Capture evidence before you optimize, exactly as the rest of the MVP loop requires (see ../understudy/SKILL.md). The decision must rest on a measured baseline, and any savings statement needs the claim.json packet that optimize-workload enforces.

Output Standard

End with:

• whether the workload was confirmed agentic, which lens applied (read-only vs state-mutating), and the fixed tool set plus any isolated implementation variable named;
• the harness id/command used (verifiers env or workflow runner);
• the objective axes (quality / latency / cost / side-effect safety where applicable), hard constraints, acceptable-regression/non-inferiority bands, and the baseline numbers;
• whether determinism was frozen (tool snapshot or seeded reset) and the holdout stayed clean;
• the interventions considered, the evidence-driven first choice, its result, and the route you would ship;
• result type: evidence-capture, evaluation, optimization-lead, heldout, or handoff;
• one recommended next command or local action.

References

• references/read-only-search.md — replayable agent harnesses, tool-output snapshotting, the env → artifact bridge, and backend-agnostic model/route comparison for read-only loops.
• references/state-mutating-workflows.md — resettable sandbox harness, final-state/policy rubric, tool-access reporting, failure-mode table, and the GEPA bridge for multi-step rollouts.