Agent Lightning
Agent optimization is a search over interacting choices. The useful question is
not which architecture is most sophisticated, but which change moves the requested
accuracy, cost, latency, and reliability frontier for this agent.
The optimizer's development budget and the resulting agent's per-run cost are
different quantities. More development budget creates room to learn; it does not
imply that the deployed agent should spend more on every task.
Your remaining development budget is reported at /artifacts/cost_budget.json
({spent, budget, remaining}), refreshed as you work. Read that file to see how
much is left, and keep iterating — measure, edit, re-score — while meaningful
budget remains; do not stop at the first plausible result. A run is finished not
because one change worked, but because further measured changes no longer improve
the frontier within the budget you still have. If remaining is large, there is
more search to do: ground more cases, probe a lever you have not tested, or add
reps to resolve a noisy comparison. Check remaining again after each expensive
step so the decision to stop is evidence-based, not a default.
Action space
| Lever |
What it changes |
Useful signal |
Main tradeoff |
| Input grounding |
Information and state visible to the model |
Relevant deployment-visible context is missing |
Longer context can distract or cost more |
| Output contract |
Representation, types, schema, files, and terminal state |
Work looks reasonable but is rejected or unreadable |
Can overfit evaluator quirks |
| Prompt |
Interpretation, priorities, and constraints |
Instructions are misunderstood or important details are ignored |
Prompt gains can be brittle |
| Tools |
Deterministic inspection, computation, and execution |
The model is approximating work a tool can do reliably |
More code and new failure modes |
| Model |
Base capability and knowledge |
The primary cannot solve grounded cases |
Cost, latency, and availability |
| Reasoning effort |
Computation used by the primary call |
Grounded hard cases remain |
Cost and latency can grow nonlinearly |
| Failure isolation |
Whether one failure damages other work |
Individual tasks crash, time out, or corrupt shared state |
Isolation does not recover the failed task |
| Conditional repair |
A second attempt informed by failure evidence |
Deployment-visible checks expose a recoverable failure |
Extra calls and possible regressions |
| Routing |
Different handling for different task classes |
Difficulty or failure risk varies predictably |
Router mistakes and operational complexity |
| Planning and interaction |
State, ordering, and tool use across multiple steps |
Long tasks lose goals or ignore observations |
More state and control-flow overhead |
| Retrieval |
Facts supplied from an available corpus |
Correct answers depend on external knowledge |
Retrieval errors and added latency |
| Critique or selection |
Additional views or candidates |
Independent attempts expose different useful information |
Multiplied calls, cost, and latency |
Reading the evidence
Different failures expose different amounts of information:
- Exceptions, missing artifacts, invalid schemas, and timeouts are objective
signals. They can support deterministic checks or focused recovery.
- A valid-looking but wrong answer may expose no label-free repair signal. More
calls with the same information can repeat the same mistake.
- Repeated failures across different attempts suggest shared blindness, a contract
mismatch, or missing capability. Diverse failures make routing, critique, or
selection more plausible.
- A development gain that disappears under validation may come from randomness,
memorized examples, training-only fields, or a different deployment path.
- An unavailable measurement is unknown evidence, not proof that a candidate
improved or regressed.
Levers interact. More reasoning cannot recover information the model never sees.
Repair cannot fix a systematic contract error when the retry receives no new
evidence. Failure isolation preserves the batch but does not repair an item. A
global model or effort increase and conditional escalation occupy different points
on the frontier.
Evaluation context
Agent evaluations are often stochastic. A score can improve while many individual
cases regress, and a single strong result can be a lucky draw. Fixed cases,
validation splits, repeated runs, frozen primary outputs, and small probes are
different ways to reduce uncertainty; their value depends on the decision and the
available budget.
Comparisons are easiest to interpret when the checkpoint, cases, model, effort,
concurrency, scorer, and execution path are held constant except for the variable
being studied. Accuracy is only one result: completion rate, downside, cost, latency,
and variance can change the decision.
Development may expose labels, metadata, or tools that do not exist at deployment.
An improvement that depends on them is not a deployed improvement. Likewise,
measurement plumbing can fail independently of the agent being tested.
Boundaries
- Preserve the target's external interface and deployment environment.
- Do not expose held-out labels or training-only answer fields to the deployed path.
- Treat the scorer and evaluation contract as immutable measurement surfaces.
- Leave a coherent measured checkpoint, not an unfinished or partially tested edit.
1---2name: agent-lightning3description: Provides the action space, tradeoffs, and evaluation context for improving an editable AI agent against a benchmark while preserving its deployment contract. Use when optimizing agent accuracy, cost, latency, or reliability.4---5
6# Agent Lightning
7
8Agent optimization is a search over interacting choices. The useful question is
9not which architecture is most sophisticated, but which change moves the requested
10accuracy, cost, latency, and reliability frontier for this agent.
11
12The optimizer's development budget and the resulting agent's per-run cost are
13different quantities. More development budget creates room to learn; it does not
14imply that the deployed agent should spend more on every task.
15
16Your remaining development budget is reported at `/artifacts/cost_budget.json`
17(`{spent, budget, remaining}`), refreshed as you work. Read that file to see how
18much is left, and keep iterating — measure, edit, re-score — while meaningful
19budget remains; do not stop at the first plausible result. A run is finished not
20because one change worked, but because further measured changes no longer improve
21the frontier within the budget you still have. If `remaining` is large, there is
22more search to do: ground more cases, probe a lever you have not tested, or add
23reps to resolve a noisy comparison. Check `remaining` again after each expensive
24step so the decision to stop is evidence-based, not a default.
25
26## Action space
27
28| Lever | What it changes | Useful signal | Main tradeoff |
29| ------------------------ | -------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------ |
30| Input grounding | Information and state visible to the model | Relevant deployment-visible context is missing | Longer context can distract or cost more |
31| Output contract | Representation, types, schema, files, and terminal state | Work looks reasonable but is rejected or unreadable | Can overfit evaluator quirks |
32| Prompt | Interpretation, priorities, and constraints | Instructions are misunderstood or important details are ignored | Prompt gains can be brittle |
33| Tools | Deterministic inspection, computation, and execution | The model is approximating work a tool can do reliably | More code and new failure modes |
34| Model | Base capability and knowledge | The primary cannot solve grounded cases | Cost, latency, and availability |
35| Reasoning effort | Computation used by the primary call | Grounded hard cases remain | Cost and latency can grow nonlinearly |
36| Failure isolation | Whether one failure damages other work | Individual tasks crash, time out, or corrupt shared state | Isolation does not recover the failed task |
37| Conditional repair | A second attempt informed by failure evidence | Deployment-visible checks expose a recoverable failure | Extra calls and possible regressions |
38| Routing | Different handling for different task classes | Difficulty or failure risk varies predictably | Router mistakes and operational complexity |
39| Planning and interaction | State, ordering, and tool use across multiple steps | Long tasks lose goals or ignore observations | More state and control-flow overhead |
40| Retrieval | Facts supplied from an available corpus | Correct answers depend on external knowledge | Retrieval errors and added latency |
41| Critique or selection | Additional views or candidates | Independent attempts expose different useful information | Multiplied calls, cost, and latency |
42
43## Reading the evidence
44
45Different failures expose different amounts of information:
46
47- Exceptions, missing artifacts, invalid schemas, and timeouts are objective
48 signals. They can support deterministic checks or focused recovery.
49- A valid-looking but wrong answer may expose no label-free repair signal. More
50 calls with the same information can repeat the same mistake.
51- Repeated failures across different attempts suggest shared blindness, a contract
52 mismatch, or missing capability. Diverse failures make routing, critique, or
53 selection more plausible.
54- A development gain that disappears under validation may come from randomness,
55 memorized examples, training-only fields, or a different deployment path.
56- An unavailable measurement is unknown evidence, not proof that a candidate
57 improved or regressed.
58
59Levers interact. More reasoning cannot recover information the model never sees.
60Repair cannot fix a systematic contract error when the retry receives no new
61evidence. Failure isolation preserves the batch but does not repair an item. A
62global model or effort increase and conditional escalation occupy different points
63on the frontier.
64
65## Evaluation context
66
67Agent evaluations are often stochastic. A score can improve while many individual
68cases regress, and a single strong result can be a lucky draw. Fixed cases,
69validation splits, repeated runs, frozen primary outputs, and small probes are
70different ways to reduce uncertainty; their value depends on the decision and the
71available budget.
72
73Comparisons are easiest to interpret when the checkpoint, cases, model, effort,
74concurrency, scorer, and execution path are held constant except for the variable
75being studied. Accuracy is only one result: completion rate, downside, cost, latency,
76and variance can change the decision.
77
78Development may expose labels, metadata, or tools that do not exist at deployment.
79An improvement that depends on them is not a deployed improvement. Likewise,
80measurement plumbing can fail independently of the agent being tested.
81
82## Boundaries
83
84- Preserve the target's external interface and deployment environment.
85- Do not expose held-out labels or training-only answer fields to the deployed path.
86- Treat the scorer and evaluation contract as immutable measurement surfaces.
87- Leave a coherent measured checkpoint, not an unfinished or partially tested edit.
Microsoft