Learning Loop

Learning loop: consolidate, absorb, refine

The engine's session-side pipeline (spec: docs/superpowers/specs/2026-08-01-learning-engine-design.md §5–§10). Capture happens automatically (plugin hooks); this skill turns what was captured into observations, and observations into reviewable skill-edit PRs. The human gate is git merge.

When to use this skill

• Promoting queue flags and completing dated learnings entries ("consolidate", a Stop-hook nudge, end of a work block).
• Drafting skill edits from ready observations ("absorb", "update my skills", "run the loop"): output is always a PR branch, never a direct edit.
• Catalog hygiene passes ("refine the skills"): prune/dedup/staleness through the same delta pipeline, report-first.
• Contested or structural edits ("experiment", "A/B this edit", "try competing fixes"): description rewrites, restructures, competing fixes where the right answer isn't obvious enough for a single draft.
• Scheduled example verification ("deep verify", "verify the examples"): promoting pinned examples/references content from status: unverified once its evals.yaml fixture passes.
• Loop health ("learning loop status"): queue depth, unabsorbed backlog, eval-suite size, ledger totals.
• For distilling external repos, use the mining skill; for authoring a new skill from scratch, skill-author.

Key directives

• Delegation posture: embed; the workflows live here; the deterministic tools live at scripts/engine/ (apply_deltas.py, run_trigger_evals.py, ledger.py, mine_transcripts.py, skill_metrics.py, observations.py, placement.py, run_variants.py, deep_verify.py, run_task_checks.py) and in the plugin hooks (recall).
• Scripts hold the pen. LLM roles draft deltas and diagnose; apply_deltas.py applies them (snapshot, bump, changelog, sidecars). Never hand-edit a skill during absorb; never bypass the script's refusals; a refusal is a design signal, not an obstacle.
• Consolidation never touches skills/ content. Its write surface is learnings/, learnings/observations/, and the evidence/evals sidecars; that boundary is what makes it autonomous-safe.
• Absorb consumes status: ready only. The ready bar (proof ≥ 2 | user-correction | three-part evidence | external official) is enforced by the observations lint; do not absorb around it.
• Merge is the gate. Every absorb/refine run ends in a PR with the evidence table; no agent merges to main skills/**. Mid-build sessions capture; they never edit skills directly.
• Dedup against everything seen , including absorbed and rejected observations, or judged-rejected findings reappear forever.
• One self-check round on consolidator and absorber output: re-read the draft against the source transcript window for misattribution, missed dead-ends, wrong anchors, before writing.
• Variants are deltas, never rewrites. Full-file candidate rewrites are forbidden (context collapse); a variant that apply_deltas refuses is a broken candidate, not a contender.

Quick start

Consolidate (autonomous-safe), then absorb to a PR:

# status: what's pending?
wc -l .robium/queue.jsonl                 # flags
grep -rc "status: ready" learnings/observations/*.md
uv run scripts/engine/skill_metrics.py    # catalog health

# consolidate: promote flags + complete entries + merge into observations
# (LLM workflow; see Decision guidance; writes learnings/ + sidecars only)

# absorb: draft deltas from ready observations, then:
uv run scripts/engine/apply_deltas.py deltas.yaml --dry-run   # review the report
git checkout -b loop/absorb-$(date +%F)
uv run scripts/engine/apply_deltas.py deltas.yaml
uv run skills/skill-author/scripts/validate_skills.py
uv run scripts/engine/run_trigger_evals.py --skills <touched...> \
  --flip-gate-baseline archive/<skill>/<old-version> --flip-skill <skill>
gh pr create --title "loop: absorb <topic>" --body-file report.md

Decision guidance

Consolidate (spec §6). Inputs: queue flags, miner output (scripts/engine/mine_transcripts.py over .robium/transcripts/), unconsolidated learnings entries; each resolved to its archived-transcript window and read in full context. Steps: promote flags that clear the noise bar into schema-v2 entries (verbatim text preserved); complete hand-written entries (evidence, skill tags, recurrence); merge into observations per the schema README's merge-on-same-finding and evolve-don't-overwrite rules; increment ledgers (scripts/engine/ledger.py) with sources; harvest eval cases (no-skill-fired → triggers.positive of the right skill, misfires → triggers.negative); draft the end-of-block retro for human sign-off; attribute successes (green blocks credit helpful to the anchors whose guidance shaped the actions; best-effort, neutral by default). Then the self-check round.

Absorb (spec §7.1). On ready observations: branch loop/absorb-YYYY-MM-DD; draft deltas feedback-conditioned (current SKILL.md + observation's symptom/ fix/dead-ends + smallest-edit directive + placement rule; run scripts/engine/placement.py per finding); apply via apply_deltas.py; verify (validator → trigger evals → flip gate); scoped dup check over touched skills; PR with the evidence table (per edit: skill, anchor, op, observation link, sources, eval results). See the delta-format reference for op semantics.

Refine: the five passes (see the refine-passes reference) re-armed on ledgers: prune harmful>0 ∧ helpful=0 first; dedup seeds from skill_metrics.py --dupes; staleness (90-day windows) unchanged; usage reads retro lines; growth review reads the archive. Output: retire/move/annotate deltas through the same pipeline → PR. Scoped refine after every absorb; full refine ~monthly.

Experiment (spec §9). Trigger: contested or structural edits where the right answer isn't obvious enough for a single draft: description rewrites, restructures, competing fixes. Flow: draft 2–3 feedback- conditioned DELTA variants (never full-file rewrites) against the observation, each its own deltas file, alongside the implicit baseline; scripts/engine/run_variants.py builds a whole-catalog scratch copy per candidate, applies its deltas through apply_deltas, and scores every variant deterministically: trigger pass-rate, flip-gate count, task pass-rate (only with --with-tasks, since task checks are real subprocess runs and never run silently), and token count; unless --no-llm, a blind judge (shelling to claude -p with shuffled candidate labels) offers a content pick that degrades to "skipped" on any failure rather than fabricate one. A candidate apply_deltas refuses never reaches scoring; it is reported as a build failure, a broken candidate, not a contender (the variants-are-deltas directive). Fitness ordering: trigger pass-rate desc, then task pass-rate desc (an unrun None never discriminates either side), then token count asc; leanness is the tiebreaker. Post the printed score table straight into the PR body; the "recommendation:" line is the engine's rank only; the human picks by merging the winning variant's deltas through the normal absorb path, never by the tool selecting unattended. --archive-losers then copies every non-winning variant (skill dir, deltas file, scores) to archive//variants///: branch points kept for later, not garbage. Worked walkthrough and score-table reading: the experiment-recipes reference.

Deep-verify (spec §8 layer 4). A scheduled lane, not a per-PR gate: pinned status: unverified examples/references get promoted mechanically when the skill's evals.yaml carries a tasks: fixture whose example: field names the file (skill-relative path). scripts/engine/deep_verify.py --inventory lists every unverified file across the catalog and whether it has a matching fixture (an unfixtured file is a coverage gap to close, not a failure); --run --skills <names...> actually runs each matched fixture (via the same run_task_checks primitive the task-check runner uses) and, on PASS, emits an annotate op flipping the marker to "status: verified (deep-verify: )"; nothing is written to skills/** directly, the emitted learnings/deltas/-deep-verify.yaml file goes through the normal apply_deltas review like any other delta. A FAIL is a finding for the printed report, not a fatal error: the CLI always exits 0, because a stale example is something to fix later, not a gate to block on. Run it on a schedule (per-build or monthly, alongside refine), not as part of every absorb.

Recall runs without invocation (UserPromptSubmit hook): ready observations matching the prompt inject as [robium-recall] context, citing ids. A wrong recall is capture signal: name the id and correct it; the consolidator counts it harmful.

Platform gotchas

• apply_deltas refuses an op whose archive dir already exists: that means a prior run bumped without merging. Rebase/merge the pending PR first.
• The trigger-eval judge shells to the claude CLI; offline or in CI without a key, pass --no-llm for the deterministic fallback (results are then keyword-based: good for gating, weaker for judging close calls).
• evidence.yaml and evals.yaml are engine-written; hand-edits will be overwritten and break increment audit trails.

Customization

• User tier (Phase 4): same modes with observations under .robium/ and the absorb destination an overlay under .claude/skills/; the workflows are path-parameterized, nothing else changes.
• Eval-case harvest thresholds and the recall budget are constants in the hook scripts; tune per install, not per session.

References

• references/delta-format.md: op semantics, deltas.yaml schema, refusal rules.
• references/promotion-bar.md: queue→facts→observations promotion criteria.
• references/refine-passes.md: the five hygiene passes, evidence-armed.
• references/learnings-loop.md: the pre-engine hardening process (history + the manual fallback).
• references/experiment-recipes.md: the variant A/B walkthrough, the contrastive-rollouts recipe, and when not to experiment.
• Engine tools (repo root): scripts/engine/ (apply_deltas.py, run_trigger_evals.py, ledger.py, mine_transcripts.py, skill_metrics.py, observations.py, placement.py, run_variants.py, deep_verify.py, run_task_checks.py).
• Spec: docs/superpowers/specs/2026-08-01-learning-engine-design.md §5–§10.

Changelog

• 0.2.1 (2026-08-03): style pass; removed em dashes throughout (no content changes).
• 0.2.0 (2026-08-02): experiment + deep-verify modes land (run_variants, deep_verify, task checks); experiment-recipes reference incl. contrastive-rollouts recipe (Phase 3, spec §9).
• 0.1.1 (2026-08-02): delta-format documents the learnings/deltas/ location convention (#81)
• 0.1.0 (2026-08-02): initial skill, consolidate/absorb/refine/status modes over the Phase 2b delta pipeline; absorbs skill-updater's promotion bar and skill-refiner's five passes as references (learning-engine Phase 2b, §13).