# Learning Loop

> The session-side surface of robium's learning engine: consolidate captured flags and learnings into evidence-counted observations, absorb ready observations into anchor-targeted skill-edit PRs via the deterministic delta pipeline, refine (prune/dedup/staleness) through the same pipeline, run blind variant A/B on contested edits ('experiment', 'A/B this edit', 'try competing fixes'), deep-verify unverified examples in pinned fixtures ('deep verify', 'verify the examples'), and report loop health. Use when: 'consolidate', 'absorb', 'run the loop', 'update my skills', 'absorb these learnings', 'refine the skills', 'learning loop status', end-of-block retros, promoting .robium/queue.jsonl flags, or drafting an absorb/refine PR. Everything before git merge may run autonomously; nothing lands on main skills/** without a human merge. Not for: mining external repos (mining), fresh skill authoring or the quality bar (skill-author), building robot applications (architect).

- Skill: `robium-ai/learning-loop-4` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add robium-ai/learning-loop-4`
- Raw SKILL.md: https://api.skillmd.com/api/skills/robium-ai/learning-loop-4/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: robium-ai (https://skillmd.com/u/robium-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/robium-ai/learning-loop-4

---


# Learning loop — consolidate, absorb, refine

The engine's session-side pipeline (spec: docs/superpowers/specs/2026-08-01-learning-engine-design.md
§5–§10). Capture happens automatically (plugin hooks); this skill turns what
was captured into observations, and observations into reviewable skill-edit
PRs. The human gate is git merge.

## When to use this skill

- Promoting queue flags and completing dated learnings entries ("consolidate",
  a Stop-hook nudge, end of a work block).
- Drafting skill edits from ready observations ("absorb", "update my skills",
  "run the loop") — output is always a PR branch, never a direct edit.
- Catalog hygiene passes ("refine the skills") — prune/dedup/staleness through
  the same delta pipeline, report-first.
- Contested or structural edits ("experiment", "A/B this edit", "try
  competing fixes") — description rewrites, restructures, competing fixes
  where the right answer isn't obvious enough for a single draft.
- Scheduled example verification ("deep verify", "verify the examples") —
  promoting pinned examples/references content from status: unverified
  once its evals.yaml fixture passes.
- Loop health ("learning loop status"): queue depth, unabsorbed backlog,
  eval-suite size, ledger totals.
- For distilling external repos, use the mining skill; for authoring a new
  skill from scratch, skill-author.

## Key directives

- Delegation posture: **embed** — the workflows live here; the deterministic
  tools live at scripts/engine/ (apply_deltas.py, run_trigger_evals.py,
  ledger.py, mine_transcripts.py, skill_metrics.py, observations.py,
  placement.py, run_variants.py, deep_verify.py, run_task_checks.py) and
  in the plugin hooks (recall).
- **Scripts hold the pen.** <!-- id: scripts-hold-the-pen --> LLM roles draft
  deltas and diagnose; apply_deltas.py applies them (snapshot, bump,
  changelog, sidecars). Never hand-edit a skill during absorb; never bypass
  the script's refusals — a refusal is a design signal, not an obstacle.
- **Consolidation never touches skills/ content.** <!-- id: consolidate-write-surface -->
  Its write surface is learnings/, learnings/observations/, and the
  evidence/evals sidecars — that boundary is what makes it autonomous-safe.
- **Absorb consumes status: ready only.** <!-- id: absorb-ready-only --> The
  ready bar (proof ≥ 2 | user-correction | three-part evidence | external
  official) is enforced by the observations lint; do not absorb around it.
- **Merge is the gate.** <!-- id: merge-is-the-gate --> Every absorb/refine
  run ends in a PR with the evidence table; no agent merges to main
  skills/**. Mid-build sessions capture; they never edit skills directly.
- **Dedup against everything seen** <!-- id: dedup-against-rejected --> —
  including absorbed and rejected observations — or judged-rejected findings
  reappear forever.
- **One self-check round** <!-- id: one-self-check-round --> on consolidator
  and absorber output: re-read the draft against the source transcript
  window for misattribution, missed dead-ends, wrong anchors, before writing.
- **Variants are deltas, never rewrites.** <!-- id: variants-are-deltas -->
  Full-file candidate rewrites are forbidden (context collapse); a variant
  that apply_deltas refuses is a broken candidate, not a contender.

## Quick start

Consolidate (autonomous-safe), then absorb to a PR:

```bash
# status: what's pending?
wc -l .robium/queue.jsonl                 # flags
grep -rc "status: ready" learnings/observations/*.md
uv run scripts/engine/skill_metrics.py    # catalog health

# consolidate: promote flags + complete entries + merge into observations
# (LLM workflow — see Decision guidance; writes learnings/ + sidecars only)

# absorb: draft deltas from ready observations, then:
uv run scripts/engine/apply_deltas.py deltas.yaml --dry-run   # review the report
git checkout -b loop/absorb-$(date +%F)
uv run scripts/engine/apply_deltas.py deltas.yaml
uv run skills/skill-author/scripts/validate_skills.py
uv run scripts/engine/run_trigger_evals.py --skills <touched...> \
  --flip-gate-baseline archive/<skill>/<old-version> --flip-skill <skill>
gh pr create --title "loop: absorb <topic>" --body-file report.md
```

## Decision guidance

**Consolidate** (spec §6) — inputs: queue flags, miner output
(scripts/engine/mine_transcripts.py over .robium/transcripts/), unconsolidated
learnings entries; each resolved to its archived-transcript window and read in
full context. Steps: promote flags that clear the noise bar into schema-v2
entries (verbatim text preserved); complete hand-written entries (evidence,
skill tags, recurrence); merge into observations per the schema README's
merge-on-same-finding and evolve-don't-overwrite rules; increment ledgers
(scripts/engine/ledger.py) with sources; harvest eval cases (no-skill-fired →
triggers.positive of the right skill, misfires → triggers.negative); draft
the end-of-block retro for human sign-off; attribute successes (green blocks
credit helpful to the anchors whose guidance shaped the actions — best-effort,
neutral by default). Then the self-check round.

**Absorb** (spec §7.1) — on ready observations: branch loop/absorb-YYYY-MM-DD;
draft deltas feedback-conditioned (current SKILL.md + observation's symptom/
fix/dead-ends + smallest-edit directive + placement rule — run
scripts/engine/placement.py per finding); apply via apply_deltas.py; verify
(validator → trigger evals → flip gate); scoped dup check over touched skills;
PR with the evidence table (per edit: skill, anchor, op, observation link,
sources, eval results). See the delta-format reference for op semantics.

**Refine** — the five passes (see the refine-passes reference) re-armed on
ledgers: prune harmful>0 ∧ helpful=0 first; dedup seeds from
skill_metrics.py --dupes; staleness (90-day windows) unchanged; usage reads
retro lines; growth review reads the archive. Output: retire/move/annotate
deltas through the same pipeline → PR. Scoped refine after every absorb; full
refine ~monthly.

**Experiment** (spec §9) — trigger: contested or structural edits where the
right answer isn't obvious enough for a single draft — description
rewrites, restructures, competing fixes. Flow: draft 2–3 feedback-
conditioned DELTA variants (never full-file rewrites) against the
observation, each its own deltas file, alongside the implicit baseline;
scripts/engine/run_variants.py builds a whole-catalog scratch copy per
candidate, applies its deltas through apply_deltas, and scores every
variant deterministically — trigger pass-rate, flip-gate count, task
pass-rate (only with --with-tasks, since task checks are real subprocess
runs and never run silently), and token count; unless --no-llm, a blind
judge (shelling to claude -p with shuffled candidate labels) offers a
content pick that degrades to "skipped" on any failure rather than
fabricate one. A candidate apply_deltas refuses never reaches scoring —
it is reported as a build failure, a broken candidate, not a contender
(the variants-are-deltas directive). Fitness ordering: trigger pass-rate
desc, then task pass-rate desc (an unrun None never discriminates either
side), then token count asc — leanness is the tiebreaker. Post the
printed score table straight into the PR body; the "recommendation:"
line is the engine's rank only — the human picks by merging the winning
variant's deltas through the normal absorb path, never by the tool
selecting unattended. --archive-losers then copies every non-winning
variant (skill dir, deltas file, scores) to
archive/<skill>/variants/<date>/<name>/ — branch points kept for later,
not garbage. Worked walkthrough and score-table reading: the
experiment-recipes reference.

**Deep-verify** (spec §8 layer 4) — a scheduled lane, not a per-PR gate: pinned
status: unverified examples/references get promoted mechanically when the
skill's evals.yaml carries a tasks: fixture whose example: field names the
file (skill-relative path). scripts/engine/deep_verify.py --inventory
lists every unverified file across the catalog and whether it has a
matching fixture (an unfixtured file is a coverage gap to close, not a
failure); --run --skills <names...> actually runs each matched fixture
(via the same run_task_checks primitive the task-check runner uses) and,
on PASS, emits an annotate op flipping the marker to
"status: verified <date> (deep-verify: <task>)" — nothing is written to
skills/** directly, the emitted learnings/deltas/<date>-deep-verify.yaml
file goes through the normal apply_deltas review like any other delta. A
FAIL is a finding for the printed report, not a fatal error — the CLI
always exits 0, because a stale example is something to fix later, not a
gate to block on. Run it on a schedule (per-build or monthly, alongside
refine), not as part of every absorb.

**Recall** runs without invocation (UserPromptSubmit hook): ready
observations matching the prompt inject as [robium-recall] context, citing
ids. A wrong recall is capture signal — name the id and correct it; the
consolidator counts it harmful.

## Platform gotchas

- apply_deltas refuses an op whose archive dir already exists — that means a
  prior run bumped without merging. Rebase/merge the pending PR first.
- The trigger-eval judge shells to the claude CLI; offline or in CI without
  a key, pass --no-llm for the deterministic fallback (results are then
  keyword-based — good for gating, weaker for judging close calls).
- evidence.yaml and evals.yaml are engine-written; hand-edits will be
  overwritten and break increment audit trails.

## Customization

- User tier (Phase 4): same modes with observations under .robium/ and the
  absorb destination an overlay under .claude/skills/ — the workflows are
  path-parameterized, nothing else changes.
- Eval-case harvest thresholds and the recall budget are constants in the
  hook scripts; tune per install, not per session.

## References

- `references/delta-format.md` — op semantics, deltas.yaml schema, refusal rules.
- `references/promotion-bar.md` — queue→facts→observations promotion criteria.
- `references/refine-passes.md` — the five hygiene passes, evidence-armed.
- `references/learnings-loop.md` — the pre-engine hardening process (history + the manual fallback).
- `references/experiment-recipes.md` — the variant A/B walkthrough, the
  contrastive-rollouts recipe, and when not to experiment.
- Engine tools (repo root): scripts/engine/ — apply_deltas.py,
  run_trigger_evals.py, ledger.py, mine_transcripts.py, skill_metrics.py,
  observations.py, placement.py, run_variants.py, deep_verify.py,
  run_task_checks.py.
- Spec: docs/superpowers/specs/2026-08-01-learning-engine-design.md §5–§10.

## Changelog

- 0.2.0 (2026-08-02): experiment + deep-verify modes land (run_variants, deep_verify, task checks); experiment-recipes reference incl. contrastive-rollouts recipe (Phase 3, spec §9).
- 0.1.1 (2026-08-02): delta-format documents the learnings/deltas/ location convention (#81)
- 0.1.0 (2026-08-02): initial skill — consolidate/absorb/refine/status modes
  over the Phase 2b delta pipeline; absorbs skill-updater's promotion bar and
  skill-refiner's five passes as references (learning-engine Phase 2b, §13).