Generating mod learning envs
Use this skill to create JSON envs consumed by /mods learn --env=<path> or bun scripts/mod-learning/learn-mod.ts --env <path>. An env describes the mod behavior to learn and the scenario-suite eval used to score candidates.
Workflow
- Define the behavior and eval before writing JSON.
- What should the mod do? Tool, turn event, tool event, provider, command, status, etc.
- What would a placebo/no-op mod fail?
- What unique sentinel strings make success unambiguous?
- Choose a path:
- Repo example:
docs/examples/mods/learning/<slug>.env.json
- Local/private: any user-requested path
- Draft strict JSON. Start from
assets/mod-learning-env.template.json if useful. No comments or trailing commas.
- Prefer
evaluation.scenarios with at least:
- happy path
- discrimination/exact-target path
- negative control
- Validate:
bun src/skills/builtin/generating-mod-envs/scripts/validate-mod-env.ts path/to/env.json
If this skill is installed outside the source tree, run the same script from this skill directory: scripts/validate-mod-env.ts.
- If asked to run it:
/mods learn --env=path/to/env.json --model=auto --backend=api --out=/tmp/<slug>-learn
The raw scripts/mod-learning/learn-mod.ts dev script detaches by default. Add --foreground only when a blocking pass/fail exit code is needed.
Use single-line --flag=value commands for TUI instructions.
Env shape
Required top-level fields:
name: human display name.
slug: stable kebab-case run/candidate slug.
objective: one-paragraph target for the generation agent.
requirements: concrete pass/fail behavior constraints.
evaluation: either a single prompt eval or a scenario suite.
Common optional fields:
targetModName: display metadata for the intended mod filename. The harness still chooses the candidate filename from slug unless --candidate-file-name is passed.
candidateDiversityHints: strategies assigned across multi-candidate runs.
modApiHints: concise API reminders that prevent bad generated code.
examples: small input/expected demos for the generation prompt.
Evaluation fields:
evaluation.outputFormat: use stream-json when checking trace markers.
evaluation.timeoutMs, evaluation.maxTurns: per-scenario defaults.
evaluation.memoryFiles: files seeded under eval $MEMORY_DIR.
evaluation.scenarios[]: scenario-specific overrides and fixtures.
- In scenario-suite envs, do not add a top-level
evaluation.prompt unless that prompt must run for every scenario. Assertion-only scenarios should have assertions and no prompt; only scenarios that require model behavior should define scenario.prompt.
requiredResultMarkers: literal strings required in the final answer.
requiredTraceMarkers: literal strings required in raw stdout/stderr.
forbiddenResultMarkers: final-answer strings that fail the run.
forbiddenTraceMarkers: raw trace strings that fail the run.
Quality rules
- Design the eval first. A useful env distinguishes success from a no-op mod.
- Use unique sentinels, e.g.
MY-MOD-CANARY-OK, not common phrases.
- Seed
memoryFiles rather than depending on real user memory or repo files.
- Include negative controls for non-use. If behavior should be conditional, verify it stays silent when not triggered.
- Include discrimination scenarios when paths, IDs, or sources matter. Put a tempting wrong sentinel in an irrelevant fixture and forbid it in the final answer.
- Put load failures in
forbiddenTraceMarkers, usually:
[mods] failed to load
[extensions] failed to load
loaded 0 mod(s)
loaded 0 extension(s)
- For eval-facing tools, require
requiresApproval: false, parallelSafe: true, and a strict no-argument schema when applicable.
- Avoid over-brittle trace markers. Prefer stable substrings like the tool name plus
"message_type":"tool_return_message".
- Keep requirements behavioral; put fragile implementation details in
modApiHints only when needed.
Minimal scenario-suite example
{
"name": "Hello tool mod learner demo",
"slug": "hello-tool",
"objective": "Learn a trusted local mod that registers a read-only hello_mod_ping tool returning a fixed sentinel.",
"requirements": [
"Register a tool named hello_mod_ping.",
"The tool must accept no parameters, require no approval, be parallelSafe, and return the exact string HELLO-MOD-OK."
],
"candidateDiversityHints": [
"Use the smallest possible tool-only implementation.",
"Add explicit defensive checks around the tool schema."
],
"modApiHints": [
"Use export function activate(letta) or a default export.",
"Use letta.tools.register({ name, description, parameters, requiresApproval, parallelSafe, run }).",
"A no-argument tool schema is { \"type\": \"object\", \"properties\": {}, \"additionalProperties\": false }."
],
"evaluation": {
"outputFormat": "stream-json",
"timeoutMs": 900000,
"maxTurns": 6,
"forbiddenTraceMarkers": ["[mods] failed to load", "loaded 0 mod(s)"],
"scenarios": [
{
"name": "happy-path",
"prompt": "Call the hello_mod_ping tool, then answer with the exact text HELLO-MOD-OK.",
"requiredResultMarkers": ["HELLO-MOD-OK"],
"requiredTraceMarkers": ["hello_mod_ping", "\"message_type\":\"tool_return_message\""]
},
{
"name": "negative-control",
"prompt": "Answer without calling tools: what is 2 + 2?",
"requiredResultMarkers": ["4"],
"forbiddenTraceMarkers": ["hello_mod_ping"]
}
]
}
}
1---2name: generating-mod-envs3description: Generates and reviews mod learning env JSON files for Letta Code local mods. Use when asked to teach, learn, or optimize a mod behavior; create, draft, validate, improve, or explain envs for `/mods learn --env`; or design evaluation scenarios, memory fixtures, requiredResultMarkers, requiredTraceMarkers, negative controls, and candidate diversity hints.4disable-model-invocation: true5user-invocable: true6metadata:7 source: https://skillmd.com/skills/letta-ai/generating-mod-envs8---910# Generating mod learning envs1112Use this skill to create JSON envs consumed by `/mods learn --env=<path>` or `bun scripts/mod-learning/learn-mod.ts --env <path>`. An env describes the mod behavior to learn and the scenario-suite eval used to score candidates.1314## Workflow15161. Define the behavior and eval before writing JSON.17 - What should the mod do? Tool, turn event, tool event, provider, command, status, etc.18 - What would a placebo/no-op mod fail?19 - What unique sentinel strings make success unambiguous?202. Choose a path:21 - Repo example: `docs/examples/mods/learning/<slug>.env.json`22 - Local/private: any user-requested path233. Draft strict JSON. Start from `assets/mod-learning-env.template.json` if useful. No comments or trailing commas.244. Prefer `evaluation.scenarios` with at least:25 - happy path26 - discrimination/exact-target path27 - negative control285. Validate:2930```bash31bun src/skills/builtin/generating-mod-envs/scripts/validate-mod-env.ts path/to/env.json32```3334If this skill is installed outside the source tree, run the same script from this skill directory: `scripts/validate-mod-env.ts`.35366. If asked to run it:3738```text39/mods learn --env=path/to/env.json --model=auto --backend=api --out=/tmp/<slug>-learn40```4142The raw `scripts/mod-learning/learn-mod.ts` dev script detaches by default. Add `--foreground` only when a blocking pass/fail exit code is needed.4344Use single-line `--flag=value` commands for TUI instructions.4546## Env shape4748Required top-level fields:4950- `name`: human display name.51- `slug`: stable kebab-case run/candidate slug.52- `objective`: one-paragraph target for the generation agent.53- `requirements`: concrete pass/fail behavior constraints.54- `evaluation`: either a single prompt eval or a scenario suite.5556Common optional fields:5758- `targetModName`: display metadata for the intended mod filename. The harness still chooses the candidate filename from `slug` unless `--candidate-file-name` is passed.59- `candidateDiversityHints`: strategies assigned across multi-candidate runs.60- `modApiHints`: concise API reminders that prevent bad generated code.61- `examples`: small input/expected demos for the generation prompt.6263Evaluation fields:6465- `evaluation.outputFormat`: use `stream-json` when checking trace markers.66- `evaluation.timeoutMs`, `evaluation.maxTurns`: per-scenario defaults.67- `evaluation.memoryFiles`: files seeded under eval `$MEMORY_DIR`.68- `evaluation.scenarios[]`: scenario-specific overrides and fixtures.69- In scenario-suite envs, do not add a top-level `evaluation.prompt` unless that prompt must run for every scenario. Assertion-only scenarios should have `assertions` and no `prompt`; only scenarios that require model behavior should define `scenario.prompt`.70- `requiredResultMarkers`: literal strings required in the final answer.71- `requiredTraceMarkers`: literal strings required in raw stdout/stderr.72- `forbiddenResultMarkers`: final-answer strings that fail the run.73- `forbiddenTraceMarkers`: raw trace strings that fail the run.7475## Quality rules7677- Design the eval first. A useful env distinguishes success from a no-op mod.78- Use unique sentinels, e.g. `MY-MOD-CANARY-OK`, not common phrases.79- Seed `memoryFiles` rather than depending on real user memory or repo files.80- Include negative controls for non-use. If behavior should be conditional, verify it stays silent when not triggered.81- Include discrimination scenarios when paths, IDs, or sources matter. Put a tempting wrong sentinel in an irrelevant fixture and forbid it in the final answer.82- Put load failures in `forbiddenTraceMarkers`, usually:83 - `[mods] failed to load`84 - `[extensions] failed to load`85 - `loaded 0 mod(s)`86 - `loaded 0 extension(s)`87- For eval-facing tools, require `requiresApproval: false`, `parallelSafe: true`, and a strict no-argument schema when applicable.88- Avoid over-brittle trace markers. Prefer stable substrings like the tool name plus `"message_type":"tool_return_message"`.89- Keep requirements behavioral; put fragile implementation details in `modApiHints` only when needed.9091## Minimal scenario-suite example9293```json94{95 "name": "Hello tool mod learner demo",96 "slug": "hello-tool",97 "objective": "Learn a trusted local mod that registers a read-only hello_mod_ping tool returning a fixed sentinel.",98 "requirements": [99 "Register a tool named hello_mod_ping.",100 "The tool must accept no parameters, require no approval, be parallelSafe, and return the exact string HELLO-MOD-OK."101 ],102 "candidateDiversityHints": [103 "Use the smallest possible tool-only implementation.",104 "Add explicit defensive checks around the tool schema."105 ],106 "modApiHints": [107 "Use export function activate(letta) or a default export.",108 "Use letta.tools.register({ name, description, parameters, requiresApproval, parallelSafe, run }).",109 "A no-argument tool schema is { \"type\": \"object\", \"properties\": {}, \"additionalProperties\": false }."110 ],111 "evaluation": {112 "outputFormat": "stream-json",113 "timeoutMs": 900000,114 "maxTurns": 6,115 "forbiddenTraceMarkers": ["[mods] failed to load", "loaded 0 mod(s)"],116 "scenarios": [117 {118 "name": "happy-path",119 "prompt": "Call the hello_mod_ping tool, then answer with the exact text HELLO-MOD-OK.",120 "requiredResultMarkers": ["HELLO-MOD-OK"],121 "requiredTraceMarkers": ["hello_mod_ping", "\"message_type\":\"tool_return_message\""]122 },123 {124 "name": "negative-control",125 "prompt": "Answer without calling tools: what is 2 + 2?",126 "requiredResultMarkers": ["4"],127 "forbiddenTraceMarkers": ["hello_mod_ping"]128 }129 ]130 }131}132```