Learn: harness retrospective + wiki knowledge harvest
SEAT NOTE (wsl claude cli seat, port 2026-10-17): the Hermes deprecation below does NOT apply on this seat - the Hermes
memory/session_searchtools it routes to do not exist here, and the close-out chain (offensive.py closeout, the close-out Stop hook) still names and mandates this skill. Run the full workflow below. Hermes-specific rows translate as: cross-session facts -> project memory (CLAUDE.md/ auto-memory); "have we hit this before" -> transcript recall pluswikiskill queries.
DEPRECATED (2026-10-11, operator directive): Hermes covers this skill's job natively. Route the work as follows and do NOT run this file's workflow wholesale:
- Phase 0 (harness retro) ->
Skill(engagement-retro)(Hermes-side skill: same drift -> root-cause -> fix loop, writes via skill_manage + memory instead of hand-edited prose).- Cross-session facts -> Hermes
memorytool (auto-loaded every session; replaces the reread-harness-retro.md discipline).- "Have we hit this before" -> Hermes
session_search(replaces failures.md archaeology).- Wiki knowledge delta (Phases 1-7) -> STILL VALID: keep the wiki-stage.py -> wiki-promote.py leak-gate mechanics below; that pipeline remains the only path for engagement-derived content into the canonical wiki.
The close-out chain (
offensive.py closeout) still nameslearn; on Hermes, runSkill(engagement-retro)for Phase 0 and the wiki-stage/promote commands directly for the delta. This file remains as the mechanics reference for that pipeline.
(Original ZCode-era workflow preserved below for the gate mechanics.)
The close-out retrospective, in two phases:
- Phase 0 (process): diff how the engagement was EXECUTED against the discipline the skills/hooks prescribe, find where the operator (you) or the harness drifted, and fix the harness so the next box does not repeat it. This is the "improve the harness on it" half.
- Phases 1-7 (knowledge): read the WHOLE finished engagement, diff it against the existing wiki, and promote only the DELTA as durable, generic knowledge. The safety net that catches everything not captured live during the box.
The client-data boundary is enforced by code, not by this prose: engagement-derived
content reaches wiki/ ONLY through wiki-stage.py -> wiki-promote.py, which runs
check-leaks.sh and fails closed. This skill READS the engagement and WRITES only
generic knowledge; it never edits the engagement's own files.
When it fires
- At close-out: run
Skill(learn)once the engagement is marked## STATUS: SOLVED(OWNED/ROOTED/COMPLETE) AND itswalkthrough.mdis assembled (the AGENTS.md execution-loop close-out step). It self-clears the moment this skill writes<eng>/.learn-done. - On demand: "extract learnings", "what did we learn", "harvest lessons", "distill", "post-mortem into the wiki" - the same steps apply.
Phase 0: Process retrospective (harness self-improvement) - DO FIRST
Before harvesting knowledge, ask: did we execute the way the skills/hooks told us to, and where we did not, whose fault is it - mine, or the harness's for not catching me? The point is not self-flagellation; it is to turn each drift into a concrete harness change so the next engagement cannot repeat it.
0a. Diff execution against discipline
Re-read the skills that governed this engagement (ctf-box, the hunt-* used, the AGENTS.md
execution loop) and check each mandated step against what actually happened on disk:
| Discipline it prescribes | How to check it was (not) done | Common drift |
|---|---|---|
| Recon tooling complete (nmap AND ffuf AND nuclei-read for web) | ls targets/$ENG/recon/*.png, .recon-tools marker, tmux window names, log.md |
ffuf never run; nuclei launched but output never read |
| Screenshot EVERY finding as it lands | count deliberate poc/*.png vs findings in Killchain.md; .screenshot-nudged |
shots only at the very end; transient states lost |
| Wiki-first before exploiting each fingerprinted service | wiki queries in transcript / log.md |
jumped to exploitation from memory |
| A hook nudge fired -> was it acted on? | grep the transcript for a nudge (e.g. "switch to ffuf") whose action never followed | nudge ignored under momentum |
| State-first / capture-as-you-go | state.md/loot.md updated mid-box vs all-at-end |
prose-in-chat lost |
If the engagement ran under the offensive driver (Skill(offensive), scripts/offensive.py), read the
state files it wrote - they say precisely where the driver fought the operator, no guessing:
Approach.md's 4a board:[x]closed vs[!]dead rows = the exhaustion-to-finding ratio; a board thick with[!]before a single[x]is the retro's starting point.Deadends.md: everyasset x classvector the driver killed (and why).decisions.md: the parked / out-of-envelope rows (offensive.py done --park), i.e. where the loop had to defer instead of push.
Cross-check those against log.md for any move made OFF the board (improvisation the driver never sequenced).
For each drift, name the root cause honestly: a skill that under-specified, a hook that should have fired a reflex but did not (or fired and was ignorable), or a me failure the harness had no mechanism to catch.
0b. Turn each drift into a harness change
Route each finding to its fix TARGET and make the change (small+clear -> apply now with the normal edit+test loop; larger -> record as a proposed item for operator approval):
- Reflex gap (a capture/coverage signal the harness could have caught) -> add/extend a hook
in
skills/hooks/(stay within "reflexes = capture/route", NOT methodology), with a test intests/. Runpython3 -m pytest tests/ -qgreen before moving on. - Discipline under-specified -> tighten the governing skill (
ctf-box/hunt-*): make the step an explicit ordered checklist item, not buried prose. Skill bodies carry lesson PROSE +[[wikilink]]ONLY - never inline runnable exploit/rootkit/PoC code (magic-signal dumps, syscall PoCs, CVE exploits, reversing walks). That code is the KNOWLEDGE half: promote it to awiki/page through the Phase 5-6 gate and point the skill at it with a one-line "try X (see [[page]])". A skill body loads wholesale into model/API context on EVERY invocation, so raw offensive code there trips the model API safety classifier (real: ctf-box's LKM-rootkit + CVE-PoC lessons got flagged on load, forcing a manual purge) and bloats context. Prose that says WHAT to try in WHAT order belongs in the skill; the code that does it belongs in the wiki. - Script/analyzer gap -> extend
scripts/(coverage, next_move, find-lint) to surface it.
0c. Log the retrospective (generic, tracked)
Append a dated section to docs/superpowers/harness-retro.md (create if missing): the drifts
found, their root cause, and the harness change made or proposed. This doc is about the HARNESS,
so it is generic and tracked - describe failures generically ("on a web box ffuf was skipped"),
never with client host/IP/cred.
Naming the engagement in this tracked file: use its dir name ONLY when that name is a neutral
codename (<platform>_<boxname>). Many engagement dirs are named after the target itself (a host, a
product, a client short-name) - writing one of those into harness-retro.md publishes a client
marker, and check-leaks.sh derives its marker list from targets/ dir names, so it WILL fail the
gate. When the dir name is target-derived, describe the engagement generically instead ("a Symfony
API target", "a single-host web bug-bounty campaign") and name nothing. Run bash scripts/check-leaks.sh after writing the retro; a FAIL here almost always means the dir name leaked.
0d. Fill the engagement's agent eval (per-engagement, gitignored)
targets/$ENG/eval.md has two halves - auto (real data) and judgement (you). Scaffolded upfront
from _eval.md for pentest/bugbounty; for ctf it self-creates the moment the next step
(eval_metrics.py --write, already run once by close-out.py at SOLVED) writes to it, so it may not
exist yet if you are running this by hand before close-out fired:
Auto first - REAL numbers, no estimation:
python3 scripts/eval_metrics.py $ENG --write
This injects a ## Metrics (auto) block from the hook telemetry (.events.jsonl) + the session
transcript recorded in .metrics.json: exact skill / hook / tool call counts, auto-detected drift
signals (every scope-guard block, state-discipline nudge, leak-warn), the start->finish time delta +
idle-filtered active time, and the box's token usage (windowed to its active period). NEVER hand-type
these - the transcript is ground truth (a real box's true output was ~12x an earlier hand-estimate).
Then the judgement half - only you can write it: in the same file, fill the Drift moments (what
the auto drift-count can't say: WHY each drift happened + the one-line fix), What went right, and the
Scores. The auto block counts drift; you narrate its cause. Dead-ends come from Deadends.md.
Unlike the retro (generic, tracked), eval.md is per-engagement and may name specifics (gitignored targets/).
Only after Phase 0 is done, proceed to the knowledge harvest below.
Steps (Phases 1-7: wiki knowledge harvest)
1. Resolve the engagement
ENG=$(cat targets/active.md)
TYPE=$(grep -m1 engagement_type targets/$ENG/state.md | cut -d: -f2 | tr -d ' ')
Confirm close-out (## STATUS: heading in state.md) or that the operator asked
explicitly. Do not harvest an in-flight engagement unless asked.
2. Inventory what the engagement taught
Read the full engagement and list every candidate GENERIC lesson:
state.md- tech/service/version that mattered and how it was handled.loot.md- default or vendor-known creds (NOT client-set passwords).Killchain.md+walkthrough.md- the chain that actually worked, with exact commands.Deadends.md- what failed. Negative knowledge is reusable: a bypass that does NOT work on tech X, a default cred changed in vendor version Y, a false-positive pattern.Vuln-index.md/Vulns/- findings and their reusable exploitation technique.- A small indicator that would have routed you to the vector early (a sink call, a token shape,
an error signature, a header quirk) -> a HIGH-PRECISION row in
scripts/tells.json(the tracked tell battery; hunt-core runs it before big reads). Ask per finding: "what one string, grepped early, names this class?" - if the answer exists and is precise, add the row now. Approach.md,log.md- anything else non-obvious that recurs.
A candidate is worth harvesting only if it is REUSABLE on the next engagement. Skip one-off client trivia.
3. Generalize + strip client specifics
Rewrite each candidate to its generic form: product + technique/cred/endpoint +
impact. Drop every client host, IP, domain, and client-set credential value. Client
specifics stay under targets/<eng>/. (The leak gate will refuse anything that slips
through, but strip up front - do not lean on the gate.)
Compact to the DELTA, not a re-explanation. A staged candidate is the MINIMAL new reusable primitive + its one gotcha (a technique delta is typically < ~15 lines), NOT a re-teach of what the target page already covers. Before promote, re-read the target section (Phase 4) and cut every line already there -- keep ONLY what the wiki did not have. A verbose dump that repeats known material is a failed harvest even if it promotes.
Format of the staged body (it becomes the wiki page verbatim): no em-dashes (comma, semicolon,
or rewrite; legal only inside a fence where the command itself contains one), and every code fence
language-tagged to match its content (sh, powershell, c, ruby, py, js, json, html;
text for prose payloads and value blocks) -- never a bare ``` fence.
Anonymization protocol (verbatim token set; the promote gate enforces the IP half):
- target/victim IPs ->
{target_ip}/{victim_ip}; domains ->{target_domain}/{victim_domain} - credentials ->
{username}/{password}/{hash}; tokens and API keys ->{token}/{api_key} - ports ->
{port}ONLY when non-standard (keep 80, 443, 22 literal - standard ports carry context) - keep the exploitation context: a placeholder must leave the technique replayable, not strip the how
4. Dedup against the wiki (the skip rule)
For each generic candidate, search the wiki first:
mcp__wiki-search__query # sub-queries: lex = keyword (tool name, CVE id,
# payload string), vec = semantic (concept /
# technique / intent), hyde = hypothetical answer
mcp__wiki-search__get / multi_get # fetch a known page by path
Find the home page (one class = one page). Read only its frontmatter and the relevant section. If the technique/payload/cred is ALREADY covered there, SKIP it - wiki has it. Keep only the delta. This step is the whole point: "extract stuff we haven't had yet".
4.5. Vector-workflow completeness check
For each Approach.md 4a board row this engagement actually populated whose vuln class is a
scripts/offensive.py VECTOR_CLASSES member (content-discovery/js-extract/recon-nuclei/
recon-nikto/wpscan-scan/osint-subdomain/osint-leaks/linux-svc-enum/ad/windows), look up its
mapped tool via CLASS_TOOL and check wiki/techniques/<area>/ (same vuln-class -> area table
as Step 5 below) for a page that actually covers that tool/technique. If missing or thin, treat it
as a candidate: generalize it the same way as Step 3 and stage a gap-fill note through the SAME
wiki-stage.py -> wiki-promote.py leak-gated pipeline used for every other candidate -- never a
direct write. Silent no-op if this engagement's board never touched a vector-baseline row (most
pre-vector-workflow engagements, or a box with no matching vector).
This runs the wiki-completeness check FOR the vector(s) this engagement actually exercised, not a
standing full-vault audit of all four vectors every time -- a web-heavy box checks web's
coverage, an AD box checks active-directory's, etc.
5. Route each survivor through stage
Never hand-edit wiki/ with engagement-derived content. Stage it.
If the lesson traces back to a vector-workflow baseline row (check the finding's originating
board row's vuln class cell against scripts/offensive.py's VECTOR_CLASSES), pick <area>
deterministically from this table instead of guessing:
| vuln class (board row) | <area> |
|---|---|
osint-subdomain, osint-leaks |
osint |
content-discovery, js-extract, recon-nuclei, recon-nikto, wpscan-scan |
web |
ad, windows (via the ad_windows vector match) |
active-directory |
linux-svc-enum |
linux |
Falls back to normal ad hoc judgment (search the wiki, pick the closest existing
techniques/<area>/ directory) for anything that isn't a vector-baseline class.
# default / vendor-known credential -> the cred cheatsheet
python3 scripts/wiki-stage.py --kind default-cred --slug <product>-default \
--body '| <product> | <version> | <user> | <pass> | observed | <generic note> |'
# reusable request / payload pattern -> the api-request cheatsheet
python3 scripts/wiki-stage.py --kind api-pattern --slug <product>-<endpoint> \
--body '| <product> | <endpoint> | <method> | <request/payload> | <auth> | <reveals> |'
# technique / bypass / tool-gotcha that ENRICHES an existing page
python3 scripts/wiki-stage.py --kind technique --slug <slug> \
--target-page techniques/<area>/<page>.md
# then edit targets/$ENG/wiki-candidates/<slug>.md to APPEND the generic '## Heading' body
# (append below the staged frontmatter -- do NOT overwrite the file, or you drop `status: pending`
# and `wiki-promote --list` will show nothing to promote)
Genuinely new class with no home page: wiki-promote merges into an EXISTING
page and skips a missing target. So first create a CONTENT-FREE generic scaffold
(frontmatter per docs/page-types.md, zero engagement data), then stage the body
against it so the substance still arrives through the gate:
# 1. scaffold wiki/techniques/<area>/<slug>.md: FRONTMATTER ONLY, no section-heading stubs
# (promote APPENDS the staged body; pre-written heading stubs are left in place and
# duplicate the real headings -- a scaffold with stubs ships a page whose top is
# a stack of empty headings)
# 2. stage the generic body (it carries ALL the '## ' headings):
python3 scripts/wiki-stage.py --kind technique --slug <slug> \
--target-page techniques/<area>/<slug>.md
# 3. after promote, verify the new page: no duplicated/empty '## ' headings at the top
For an external source that also informs the lesson (a CVE writeup, an advisory), hand
that part to Skill(research-ingest) rather than duplicating it here.
6. Promote through the leak gate
python3 scripts/wiki-promote.py --list # review pending candidates
python3 scripts/wiki-promote.py --review <slug> # read one in full
python3 scripts/wiki-promote.py --promote all # leak-checked merge + re-index
Promote runs check-leaks.sh on each body and refuses (writes nothing) on a client
marker. Report what promoted, what was refused, and why. A zero-yield harvest is worth
a sanity check (was the sweep actually run, or is the wiki already saturated for this box).
7. Re-index, lint, self-clear
python3 scripts/lint-wiki.py -q # must be clean (broken links, stale index)
# only if a NEW page was created:
python3 scripts/gen_index.py && python3 scripts/build_moc.py && qmd update && qmd embed
# GATE: do NOT self-clear with eval.md's judgement half blank (Phase 0d). This exits 1 and
# refuses the marker until Drift moments / What went right / Score are filled -- learn self-cleared
# with the human half empty on a real box, so this is enforced, not advisory.
python3 scripts/eval_metrics.py $ENG --check-judgement || { echo "fill eval.md judgement half first"; exit 1; }
touch targets/$ENG/.learn-done # marks this engagement's learn pass done
Log one GENERIC line to session/log.md (e.g. "learn: promoted 3 -> jwt-attacks,
default-credentials") and an audit line to targets/$ENG/log.md. Never put a client
host/IP/domain in either.
Guardrails
- Generic only, gate-enforced. All engagement-derived writes go stage -> promote; the leak check is the code boundary. A new page is a content-free scaffold; its substance still comes through the gate.
- Dedup first. Enrich the existing page; do not create a second page for a class
wiki already covers.
qmdbefore staging, every time. - No fabrication. Harvest only what the engagement files actually record. If a lesson is not backed by the state/loot/Killchain/walkthrough/Deadends record, drop it.
- Read-only on the engagement. Phase 0 may edit HARNESS files (
skills/,skills/hooks/,scripts/,tests/,docs/superpowers/harness-retro.md) and Phases 1-7 write towiki/(via the gate) and the.learn-donemarker - but NEVER the engagement's own findings/narrative. - No offensive code in a skill body. A tightened skill (
ctf-box/hunt-*) gets lesson prose +[[wikilinks]]only. Runnable exploit/rootkit/PoC code lives inwiki/(through the Phase 5-6 gate) and is referenced by link, never inlined - the skill loads into API context wholesale and the classifier flags raw offensive code (it has). Reversing walks, syscall PoCs, magic-signal dumps, CVE exploits = wiki page + a one-line skill pointer. - Phase 0 changes ship green. Any hook/script edit lands with a test and a passing
python3 -m pytest tests/ -q; do not leave the harness broken by an improvement. - Retro log is generic + tracked.
harness-retro.mddescribes harness failures generically (no client host/IP/cred); the leak boundary still applies.
Report: Phase 0 - drifts found, root cause each, harness changes made vs proposed, retro-log path, test status. Phases 1-7 - candidates found, skipped-as-already-in-wiki, staged, promoted, refused, pages touched (and any new page created), lint status.