Session Trace Mining

Session Trace Mining

Overview

Turns session transcripts into session-trace-1.0 records: one record per change that measurably moved a metric, answering which operator, what was wrong, what changed, why it worked at the machine level, the verbatim code, and the gain.

The pipeline splits three ways: scripts do deterministic extraction, an agent does semantic distillation, and sixteen mechanical gates catch fabrication. The gates are why the output can be trusted. A model filling fields is the weak link, so nothing enters the store that cannot be checked against the transcript.

What makes a transcript corpus different from a repository of commits: there is no repository to check a citation against. The transcript is the corpus. So provenance is a set name plus a set-relative path plus a per-line digest, and every number must trace to a span the packet carried — with the agent's own prose deliberately excluded from that span set.

Why every part of it lives here

The same shape of problem — extract deterministically, distil semantically, gate mechanically — applies to any corpus of optimisation history. What travels between such pipelines is design, not code: the three-way split, the declarative make_schema.py patch list, the packet contract, and the discipline of writing down every pitfall with the number behind it.

Nothing is imported. This skill is self-contained, deliberately:

Each of the three pins its own behaviour with a self_test() that runs standalone (python3 families.py, python3 score.py). score.py reproduces the curve published by the shared ranking model in tools/wiki_score.py for the inputs this corpus produces — 4% → 0.35, 20% → 0.66, 99% → 1.0, and the 0.35 feedback band — so scores stay comparable with the committed store by verification rather than by coupling. If the shared model changes, that self-test is where the divergence surfaces.

The only paths outside the skill are the two data locations declared in config.py: where the product goes (kernel_wiki/session_trace/<set>/) and the dedup / overlap scan over kernel_wiki/records. No logic crosses the boundary.

Setup

One thing has to be configured, because no transcripts ship with this repository:

1. point STM_ROOT at the directory holding your own transcript archive;
2. register your sets in scripts/config.py SETS, replacing the single example-set placeholder entry. A set is named, never passed as a path, so a record's provenance survives moving the archive.

Both output locations are derived and need no configuration:

• scratch — /tmp/session-trace-mining/<set>/: parsed candidates, segments, packets. Reproducible, never committed. Override with STM_WORKSPACE.
• product — <gpu-wiki>/kernel_wiki/session_trace/<set>/: records and the reports that justify them. Override with STM_STORE. It sits beside the committed store but not inside kernel_wiki/records/: these records use the derived session-trace-1.0 schema, so filing them into the committed store would make that store fail its own schema gate. Promoting one means rewriting it against schema/kernel/schema.json, deliberately and one at a time.

Plain python3 is enough. The schema gate additionally needs jsonschema; without it that one gate reports SKIP and the other fifteen still run.

Pipeline

S=<skill-dir>/scripts
export STM_ROOT=/path/to/your/transcript-archive
export STM_SET=example-set             # a key of config.SETS -- register your own

python3 $S/make_schema.py --check      # schema matches its patch list
python3 $S/ingest.py                   # transcripts -> work/versions.jsonl
python3 $S/recon.py                    # -> reports/recon.md   READ THIS FIRST
python3 $S/partition.py                # -> work/segments.jsonl, reports/partition.md
python3 $S/build_packets.py            # -> packets/<seg>.{json,diff}
# distil (see below), then:
python3 $S/validate_store.py --verbose          # sixteen gates
python3 $S/validate_store.py --injection-tests
python3 $S/score_records.py            # worth.rank.score + records/index.json
python3 $S/make_readme.py              # -> kernel_wiki/session_trace/<set>/README.md

Re-run the last two after every distillation batch.

Read reports/recon.md before distilling. It decides what the product can honestly be: the citable-number share, the diff-coverage share, and how much the committed store already covers. If a set turns out to have almost no numbers, its value is mechanism and anti-patterns — do not force a gain claim onto every record.

Two candidate units

The unit is a property of the corpus, not a preference, and it is declared per set in config.SETS. Getting it wrong yields either nothing or nonsense, so it was settled by a probe before any schema existed (see references/lessons.md §1–2).

Evidence tiers

The single most important design decision. Every span carries a tier, and only three of five may be cited:

T4 is excluded because admitting it makes the fabrication gate vacuous: the agent's invented number becomes its own proof. T5 is excluded because those prompts state the target percentage, which would license any number near it.

The sixteen gates

schema · ids · layout · provenance · verbatim · no-fabrication · direction · raw-isolation · relations · index · evidence-tier · diff-coverage · unit-normalization · wiki-overlap · pairing-integrity · anonymization

The six that carry the weight:

• provenance — re-resolves the set by name, the file by set-relative path, and every cited line by sha256(raw line)[:12]. Retargeting a citation fails; moving the whole archive to another absolute path still passes. A cited line with no digest is a failure too — without that clause the check silently does nothing, which is what the line-shift injection caught.
• verbatim — implementation.snippet must appear literally in packets/<seg>.diff. The gate and the distiller read the same file on purpose.
• no-fabrication — every number in worth.gain must be in the packet's evidence_text, or derivable from it by one of exactly two closed-form rules (before/after, or speedup=Nx). Derivation from arbitrary pairs of pool numbers is deliberately not allowed.
• unit-normalization — the delta must be reproducible from the measured levels, and no absolute time may appear inside worth.gain.
• wiki-overlap — nothing the committed store already covers: a colliding (operator, version) dedup key, a record id, or an episode_key. It scans kernel_wiki/records, and an empty scan is a failure, not a pass — an overlap gate whose index resolved to nothing would print OK forever.
• anonymization — the served layers must name no person, host or corpus path. Session transcripts are full of /home/<user>/..., and payload is what gets served. The record id is checked separately, because a home path flattened into a slug has no slash left for the path pattern to catch.

Never weaken a gate to make records pass, and never let a distilling agent edit scripts/. When a gate looks wrong, verify by injection: validate_store.py --injection-tests mutates a record (and, where the error lives there, its packet) and asserts that the named gate complains. Adding a gate without an injection test is how a store ends up falsely green — two of the eleven cases here were asleep on their first run.

Distillation

Spawn agents with references/distill-brief.md verbatim, substituting the placeholders. Batch by set and record type so a failure has a small blast radius, and point every agent at the one record that already passes as the worked example.

Require each agent to run validate_store.py itself and iterate to green, and to report which fields the packet was too thin to fill and which gate blocked it. That report is the main signal for improving the pipeline; treat a batch that reports no difficulties with suspicion.

When several agents write into one store concurrently, tell them explicitly to ignore gate failures naming records they do not own.

Porting to another corpus

Everything is corpus-agnostic except two places:

• scripts/config.py SETS — register the set: its path under the archive root, its transcript format, its candidate unit, and default scope. Defaults are fallbacks only; ingest.py detects hardware and DSL and records which happened in arch_basis / dsl_basis.
• scripts/transcripts.py — the only file that knows how a session log is shaped. A third agent product means one new parse_* function returning the same Event stream, plus a branch in detect_format. Everything downstream sees events and never a raw line.

Two corpus-specific details that will need attention on a new corpus: how a long-running benchmark's output is linked back to the command that launched it (in Codex logs it is a SESSION_ID=N handshake), and which label words name a whole implementation rather than a knob (partition.IMPL_LABEL_RE).

Decide the unit before writing any schema. It is what ids, pairing, dedup_key and the whole worth.gain ladder key on, so getting it wrong means re-doing the schema, the packets and three gates. Three probes exist for exactly that decision and should be re-run on a new corpus:

python3 $S/probe_versions.py <transcript> [...]   # per file: versions, edits, metrics, pairing
python3 $S/probe_set.py <set-root>                # does the ladder exist across the set?
python3 $S/probe_ab.py <set-root>                 # if not, how many measured A/Bs are there?

probe_versions.py answers "can I see versions in one file"; probe_set.py answers the question that actually matters for a ladder, since one session is one version; probe_ab.py sizes the fallback. Write the pass bars down before running them, and if a corpus fails its bar, change the unit rather than the bar.

Resources

• references/lessons.md — read before starting. Every pitfall found while building this, with the measured numbers behind each: why one session is one version, why the codex sets cannot be paired before/after, the s-for-seconds trap, the table column that inherited a unit it did not have, and the two gates that were asleep.
• references/distill-brief.md — the agent brief template.
• assets/schema/session-trace-1.0.schema.json — the record schema, generated by make_schema.py from clean-1.3.frozen.json, which is a pinned byte copy of this repository's schema/kernel/schema.json.