Playbook Codegen Rules (@ax-llm/ax)
Use this skill to generate context-playbook code. A playbook grows an evolving body of task knowledge and renders it into a program's context. The evolution engine (ACE — Agentic Context Engineering) is hidden behind playbook(...), exactly as optimize(...) hides its optimizer. Prefer the playbook(...) concept; only reach for AxACE directly when the user explicitly wants the low-level engine.
Use These Defaults
- Create with
playbook(program, { studentAI, teacherAI? }); it returns an AxPlaybook handle.
- Grow offline with
await pb.evolve(examples, metric) — returns { bestScore, playbook }.
- Grow online with
await pb.update({ example, prediction, feedback }) — no metric needed.
- Apply with
pb.applyTo(program) (defaults to the bound program).
- Persist with
pb.toJSON() and restore with playbook(program, opts).load(snapshot).
- Inspect with
pb.render() (markdown) and pb.getState() ({ playbook, artifact }).
- For agents use
agent.playbook({ target: 'actor' | 'responder' }); default target is 'actor'.
- Use a cheaper
studentAI to run the program and an optional stronger teacherAI to reflect/curate.
- Pass AI service options for the reflect/curate calls as
teacherOptions. A teacher marked isExpensive (for example OpenAI gpt-5.5-pro) is rejected unless you pass teacherOptions: { useExpensiveModel: 'yes' }. The generated Python, Java, C++, Go, and Rust packages apply the same gate and accept teacherOptions in their option maps.
- Prefer
ai(), ax(), and agent() for new code.
Critical Rules
playbook(...) binds to an AxGen program; evolve/update need that program's signature.
evolve() returns only { bestScore, playbook }. There is no Pareto front and no optimizedProgram — that is optimize(...)'s shape, not a playbook's.
update({ example, prediction, feedback }) requires the full { example, prediction }; example must match the program's input fields (plus any expected output). Do not pass bare input fields at the top level.
update() works without a prior evolve()/load() — the handle hydrates lazily on first use.
applyTo() injects a ## Context Playbook block into the program description; calling it repeatedly recomposes from the original base (no stacking).
- Keep the offline
metric deterministic and cheap, like a GEPA metric.
- A playbook is plain JSON. Persist
pb.toJSON() and load(...) it into a fresh program for production.
- The playbook engine, construction-time agent attachment, failure harvesting,
and verified agent evolution are available in TypeScript and the generated
Python, Java, C++, Go, and Rust packages. Use each package's native casing and
callback types.
Offline Pattern (evolve)
import { type AxMetricFn, ai, ax, playbook } from '@ax-llm/ax';
const program = ax('review:string -> sentiment:class "positive, negative"');
const studentAI = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY! });
const metric: AxMetricFn = ({ prediction, example }) =>
(prediction as any).sentiment === (example as any).sentiment ? 1 : 0;
const pb = playbook(program, { studentAI, maxEpochs: 2 });
const { bestScore } = await pb.evolve(train, metric);
pb.applyTo(program);
Online Pattern (update)
// After a real run, feed the outcome back so the playbook keeps learning.
await pb.update({
example: { review: 'Five stars, would buy again.' },
prediction: { sentiment: 'negative' },
feedback: 'WRONG: enthusiastic praise is positive.',
});
pb.applyTo(program);
Persist And Restore
const snapshot = pb.toJSON(); // { playbook, artifact } — plain JSON
// later, in another process / a production program instance:
playbook(prodProgram, { studentAI }).load(snapshot).applyTo(prodProgram);
Agents
a.playbook({ target }) returns an agent-aware AxAgentPlaybook (the stage AxPlaybook handle plus an agent-level evolve). The one playbook the agent renders into its prompt grows three ways:
- Continuous (trust): the construction-time
playbook option (see ax-agent) harvests each run's failures automatically — no dataset.
- On-demand (trust):
apb.update({ example, prediction, feedback }).
- Batch verified (proof):
apb.evolve(dataset, options) runs the full agent over a task set, mines failure clusters, and proposes one playbook bullet per weakness; with verify (default on) it keeps a bullet only if held-in improves AND the validation held-out set does not regress, else exact rollback. verify: false = trust-batch. Bullets-only.
const a = agent('ticket:string -> reply:string', { ai });
const apb = a.playbook({ target: 'actor' }); // agent-aware handle; 'actor' (default) or 'responder'
await apb.update({ example, prediction, feedback }); // online: injected into the live stage prompt
const result = await apb.evolve(
{ train, validation }, // AxAgentEvalDataset
{ metric, runsPerTask: 2 }, // verify:true by default
);
Each teacher call uses the teacherOptions set next to its model:
a.playbook({ teacherAI, teacherOptions }) and the construction-time playbook: { teacherAI, teacherOptions } config cover the reflector and curator calls, including the ones apb.evolve(...) makes to apply a proposal.
apb.evolve(dataset, { teacherAI, teacherOptions }) covers the weakness miner.
- Both teachers default to the agent's
judgeAI, then the student. With an isExpensive judgeAI, set teacherOptions: { useExpensiveModel: 'yes' } in both places, plus judgeOptions when the built-in judge scores the runs.
The agent-level evolve(dataset, options) is distinct from the program-level pb.evolve(examples, metric) above: it takes an AxAgentEvalDataset plus options, runs the whole pipeline, and returns baseline/final held-in & held-out with per-bullet outcomes (no { bestScore }). For full-pipeline tuning of agent instructions and demos (not the playbook) use agent.optimize(...) (GEPA).
Generated packages expose that same agent-bound loop with language-shaped APIs:
| Language |
Agent-bound evolve call |
| Python |
agent.playbook().evolve(dataset, options) |
| Java |
agent.playbook(null).evolve(dataset, options) |
| C++ |
agent.get_playbook()->evolve(dataset, options) |
| Go |
agent.GetPlaybook().EvolveAgent(ctx, dataset, options) |
| Rust |
playbook.evolve_agent(&mut agent, client, dataset, options) |
All five generated packages thread structured failureSignals through agent
evaluation predictions. The default verify gate accepts a proposed bullet only
when held-in score improves and held-out score stays within epsilon; rejection
restores the exact prior snapshot. Scoring is host-shaped: TypeScript uses its
metric, Python/Java/Go can accept a metric callback, and all generated ports can
use task score/scores values plus the agent evaluation result.
Playbook vs optimize()
playbook(...) — accumulate reusable, evolving task knowledge; the only path that also learns online via update(...).
optimize(...) / agent.optimize(...) — tune instruction text and few-shot demos offline to a best/Pareto result.
- They are complementary; a project can use both.
Troubleshooting
- "Cannot convert undefined or null to object" from
update() → you passed input fields at the top level; wrap them in example: { ... }.
- Empty playbook after
evolve() → the model already scored well, so nothing was curated; use harder/ambiguous examples or a weaker studentAI to surface lessons.
- Playbook not affecting an agent's behavior → ensure
apply is not false and you used agent.playbook(...) (not a bare playbook() on an internal program).
- Playbook never grows with an
isExpensive teacher → the reflector and curator calls are rejected before any request is sent, and only verbose: true logs it. Set teacherOptions: { useExpensiveModel: 'yes' }. In apb.evolve(...), a rejected miner call is reported as a miner failed message through onProgress (and the console with verbose: true).
See Also
ax-gepa - optimize(...) and AxGEPA for instruction/demo tuning.
ax-agent-context - choosing between contextMap, contextPolicy, agent.playbook(...), and recall.
ax-agent-optimize - agent.optimize(...) GEPA tuning for agents.
1---2name: ax-playbook3description: This skill helps an LLM generate correct playbook code using @ax-llm/ax. Use when the user asks about playbook(), AxPlaybook, context playbooks, evolving context, ACE / Agentic Context Engineering, agent.playbook(), or growing/applying task knowledge offline and online with evolve() and update().4version: "25.4.11"5metadata:6 source: https://skillmd.com/skills/ax-llm/ax-playbook7---89# Playbook Codegen Rules (@ax-llm/ax)1011Use this skill to generate context-playbook code. A playbook grows an evolving body of task knowledge and renders it into a program's context. The evolution engine (ACE — Agentic Context Engineering) is hidden behind `playbook(...)`, exactly as `optimize(...)` hides its optimizer. Prefer the `playbook(...)` concept; only reach for `AxACE` directly when the user explicitly wants the low-level engine.1213## Use These Defaults1415- Create with `playbook(program, { studentAI, teacherAI? })`; it returns an `AxPlaybook` handle.16- Grow offline with `await pb.evolve(examples, metric)` — returns `{ bestScore, playbook }`.17- Grow online with `await pb.update({ example, prediction, feedback })` — no metric needed.18- Apply with `pb.applyTo(program)` (defaults to the bound program).19- Persist with `pb.toJSON()` and restore with `playbook(program, opts).load(snapshot)`.20- Inspect with `pb.render()` (markdown) and `pb.getState()` (`{ playbook, artifact }`).21- For agents use `agent.playbook({ target: 'actor' | 'responder' })`; default target is `'actor'`.22- Use a cheaper `studentAI` to run the program and an optional stronger `teacherAI` to reflect/curate.23- Pass AI service options for the reflect/curate calls as `teacherOptions`. A teacher marked `isExpensive` (for example OpenAI `gpt-5.5-pro`) is rejected unless you pass `teacherOptions: { useExpensiveModel: 'yes' }`. The generated Python, Java, C++, Go, and Rust packages apply the same gate and accept `teacherOptions` in their option maps.24- Prefer `ai()`, `ax()`, and `agent()` for new code.2526## Critical Rules2728- `playbook(...)` binds to an `AxGen` program; `evolve`/`update` need that program's signature.29- `evolve()` returns only `{ bestScore, playbook }`. There is no Pareto front and no `optimizedProgram` — that is `optimize(...)`'s shape, not a playbook's.30- `update({ example, prediction, feedback })` requires the full `{ example, prediction }`; `example` must match the program's input fields (plus any expected output). Do not pass bare input fields at the top level.31- `update()` works without a prior `evolve()`/`load()` — the handle hydrates lazily on first use.32- `applyTo()` injects a `## Context Playbook` block into the program description; calling it repeatedly recomposes from the original base (no stacking).33- Keep the offline `metric` deterministic and cheap, like a GEPA metric.34- A playbook is plain JSON. Persist `pb.toJSON()` and `load(...)` it into a fresh program for production.35- The playbook engine, construction-time agent attachment, failure harvesting,36 and verified agent evolution are available in TypeScript and the generated37 Python, Java, C++, Go, and Rust packages. Use each package's native casing and38 callback types.3940## Offline Pattern (evolve)4142```typescript43import { type AxMetricFn, ai, ax, playbook } from '@ax-llm/ax';4445const program = ax('review:string -> sentiment:class "positive, negative"');46const studentAI = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY! });47const metric: AxMetricFn = ({ prediction, example }) =>48 (prediction as any).sentiment === (example as any).sentiment ? 1 : 0;4950const pb = playbook(program, { studentAI, maxEpochs: 2 });51const { bestScore } = await pb.evolve(train, metric);52pb.applyTo(program);53```5455## Online Pattern (update)5657```typescript58// After a real run, feed the outcome back so the playbook keeps learning.59await pb.update({60 example: { review: 'Five stars, would buy again.' },61 prediction: { sentiment: 'negative' },62 feedback: 'WRONG: enthusiastic praise is positive.',63});64pb.applyTo(program);65```6667## Persist And Restore6869```typescript70const snapshot = pb.toJSON(); // { playbook, artifact } — plain JSON71// later, in another process / a production program instance:72playbook(prodProgram, { studentAI }).load(snapshot).applyTo(prodProgram);73```7475## Agents7677`a.playbook({ target })` returns an agent-aware `AxAgentPlaybook` (the stage `AxPlaybook` handle plus an agent-level `evolve`). The one playbook the agent renders into its prompt grows three ways:7879- Continuous (trust): the construction-time `playbook` option (see `ax-agent`) harvests each run's failures automatically — no dataset.80- On-demand (trust): `apb.update({ example, prediction, feedback })`.81- Batch verified (proof): `apb.evolve(dataset, options)` runs the full agent over a task set, mines failure clusters, and proposes one playbook bullet per weakness; with `verify` (default on) it keeps a bullet only if held-in improves AND the `validation` held-out set does not regress, else exact rollback. `verify: false` = trust-batch. Bullets-only.8283```typescript84const a = agent('ticket:string -> reply:string', { ai });85const apb = a.playbook({ target: 'actor' }); // agent-aware handle; 'actor' (default) or 'responder'86await apb.update({ example, prediction, feedback }); // online: injected into the live stage prompt87const result = await apb.evolve(88 { train, validation }, // AxAgentEvalDataset89 { metric, runsPerTask: 2 }, // verify:true by default90);91```9293Each teacher call uses the `teacherOptions` set next to its model:9495- `a.playbook({ teacherAI, teacherOptions })` and the construction-time `playbook: { teacherAI, teacherOptions }` config cover the reflector and curator calls, including the ones `apb.evolve(...)` makes to apply a proposal.96- `apb.evolve(dataset, { teacherAI, teacherOptions })` covers the weakness miner.97- Both teachers default to the agent's `judgeAI`, then the student. With an `isExpensive` `judgeAI`, set `teacherOptions: { useExpensiveModel: 'yes' }` in both places, plus `judgeOptions` when the built-in judge scores the runs.9899The agent-level `evolve(dataset, options)` is distinct from the program-level `pb.evolve(examples, metric)` above: it takes an `AxAgentEvalDataset` plus options, runs the whole pipeline, and returns baseline/final held-in & held-out with per-bullet outcomes (no `{ bestScore }`). For full-pipeline tuning of agent instructions and demos (not the playbook) use `agent.optimize(...)` (GEPA).100101Generated packages expose that same agent-bound loop with language-shaped APIs:102103| Language | Agent-bound evolve call |104|---|---|105| Python | `agent.playbook().evolve(dataset, options)` |106| Java | `agent.playbook(null).evolve(dataset, options)` |107| C++ | `agent.get_playbook()->evolve(dataset, options)` |108| Go | `agent.GetPlaybook().EvolveAgent(ctx, dataset, options)` |109| Rust | `playbook.evolve_agent(&mut agent, client, dataset, options)` |110111All five generated packages thread structured `failureSignals` through agent112evaluation predictions. The default verify gate accepts a proposed bullet only113when held-in score improves and held-out score stays within `epsilon`; rejection114restores the exact prior snapshot. Scoring is host-shaped: TypeScript uses its115metric, Python/Java/Go can accept a metric callback, and all generated ports can116use task `score`/`scores` values plus the agent evaluation result.117118## Playbook vs optimize()119120- `playbook(...)` — accumulate reusable, evolving task knowledge; the only path that also learns online via `update(...)`.121- `optimize(...)` / `agent.optimize(...)` — tune instruction text and few-shot demos offline to a best/Pareto result.122- They are complementary; a project can use both.123124## Troubleshooting125126- "Cannot convert undefined or null to object" from `update()` → you passed input fields at the top level; wrap them in `example: { ... }`.127- Empty playbook after `evolve()` → the model already scored well, so nothing was curated; use harder/ambiguous examples or a weaker `studentAI` to surface lessons.128- Playbook not affecting an agent's behavior → ensure `apply` is not `false` and you used `agent.playbook(...)` (not a bare `playbook()` on an internal program).129- Playbook never grows with an `isExpensive` teacher → the reflector and curator calls are rejected before any request is sent, and only `verbose: true` logs it. Set `teacherOptions: { useExpensiveModel: 'yes' }`. In `apb.evolve(...)`, a rejected miner call is reported as a `miner failed` message through `onProgress` (and the console with `verbose: true`).130131## See Also132133- `ax-gepa` - `optimize(...)` and `AxGEPA` for instruction/demo tuning.134- `ax-agent-context` - choosing between contextMap, contextPolicy, `agent.playbook(...)`, and recall.135- `ax-agent-optimize` - `agent.optimize(...)` GEPA tuning for agents.