# Porting 2 Oracle > porting-2-oracle - Skill: `handy-computer/porting-2-oracle` (Agent Skill) - Install (CLI): `npx skillmds@latest add handy-computer/porting-2-oracle` - Raw SKILL.md: https://api.skillmd.com/api/skills/handy-computer/porting-2-oracle/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: handy-computer (https://skillmd.com/u/handy-computer) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/handy-computer/porting-2-oracle --- # porting-2-oracle Second stage of the porting pipeline. Produces the oracle packet Stage 4 implements against: tensor dumps, reference transcripts, measured reference WER, and provisional tolerances. ## Preconditions - `reports/porting///intake.json` exists, is schema-valid, Preflight Gate A green. - The per-family reference environment exists at `scripts/envs//pyproject.toml`. If not, this skill creates it. - The per-family dump adapter exists at `scripts/dump_reference__.py`. If not, this skill creates a skeleton following the contract in `scripts/lib/ref_dump.py` (or in one of the existing dumpers). - A committed reference checkpoint is available to the reference framework (HF download or `$TRANSCRIBE_MODELS_DIR//`). - `uv` is installed. ## Oracle packet contents After this stage finishes, the oracle packet is exactly: ```text tests/golden//.manifest.json build/validate/////ref/ # tensor dumps + sidecars build/validate///dump_coverage.json # tensor catalog build/validate/////ref/transcript.json # reference transcript per decode case reports/wer/-REF..jsonl # per-utterance reference hyps (one-for-one debug oracle) reports/wer/-REF..score.json # reference-measured WER baseline (the gate, not the published number) tests/tolerances/.json # provisional, Stage 4 finalizes ``` `` is the manifest case identifier (typically `jfk` for the default sample; families may add additional audio cases). Per-stage tensors live under `/encoder/`, `/decoder/`, etc., per the dumper's stage names. ## Workflow ``` Oracle progress: - [ ] Step 1: Confirm or write per-family env + dumper - [ ] Step 2: Complete the golden manifest - [ ] Step 3: Run the dumper for every manifest case - [ ] Step 4: Verify reference transcripts - [ ] Step 5: Generate dump_coverage.json - [ ] Step 6: Write provisional tolerances from per-tensor magnitude - [ ] Step 7: Reference WER baseline ran on the acceptance dataset - [ ] Step 8: Sign-off review ``` ### Step 1: Per-family env and dumper Check for `scripts/envs//pyproject.toml` and `scripts/dump_reference__.py`. **If the family is already ported** (either script exists): use as-is. **If the family is new**: - Create `scripts/envs//pyproject.toml` from the closest existing reference env. - Create `scripts/dump_reference__.py` from the closest dumper. It MUST emit tensors via `ref_dump.write_tensor` and transcripts via `write_transcript`; sidecars must include `rms` and `p99_abs`. - Surface only unresolved technical decisions to the user. ### Step 2: Complete the golden manifest (execute) `porting-1-intake` wrote a skeleton at `tests/golden//.manifest.json`. Fill it in: - `reference.{kind, entrypoint, source, revision}` — `kind` from `intake.reference_framework`, `entrypoint` pointing to the family dumper, `source` + `revision` to the pinned upstream URL + tag. - `frontend`, `tokenizer_summary`, `capabilities` from intake. - `cases: [...]` — the audio cases the dumper will run. Default is `[{audio: "samples/jfk.wav"}]`. Add additional rows only when the family has unusual audio characteristics that one sample cannot cover (non-English, tonal, music, long-form). Confirm with the user before adding cases beyond the default. - Remove the `_skeleton` marker. See `tests/golden/parakeet/parakeet-tdt-0.6b-v2.manifest.json` for the complete shape. ### Step 3: Reference dumps (execute) For every entry in `manifest.cases[]`, run the dumper's encoder + decode subcommands. Default invocation for `samples/jfk.wav`: ```bash uv run --project scripts/envs/ \ scripts/dump_reference__.py encoder \ --model \ --audio samples/jfk.wav \ --out build/validate///jfk/encoder/ref uv run --project scripts/envs/ \ scripts/dump_reference__.py decode \ --model \ --audio samples/jfk.wav \ --out build/validate///jfk/decode/ref ``` Subcommand names vary per dumper — check the script's `--help` for the exact set. For families with extra subcommands (e.g. `mel`, `prompt`) run each as the family contract requires. If the reference produces a transcript that is implausible (gibberish, empty, language mismatch), STOP — the reference setup is broken, and fixing it is a precondition for any C++ work. Do not proceed. ### Step 4: Reference transcripts (execute) Every reference decode pass should produce a transcript when the dumper exposes one. Use `scripts/lib/ref_dump.py::write_transcript` (most existing dumpers already call it). Verify `build/validate/////ref/transcript.json` exists for any decode pass that produces text or tokens. If the reference framework does not expose a transcript helper for a given decode path, skip the transcript write for that case rather than inventing one — Stage 4's behavior validation runs against the C++ runtime via the family-doc Capability Validation table, not against synthetic reference transcripts. ### Step 5: Generate dump_coverage.json (execute) Walk `build/validate///` and write `dump_coverage.json` with `{family, variant, tensors: [{case, stage_dir, stage, name, shape, dtype}, ...]}`. One entry per **tensor** sidecar — that means every `.json` file under `//ref/` whose contents include the tensor metadata fields written by `scripts/lib/ref_dump.py::write_tensor` (`name`, `shape`, `dtype`, `layout`, plus `rms` / `p99_abs`). Skip any other JSON files in the same directory: `transcript.json` (text/tokens, not a tensor), and any future behavioral artifacts the dumper grows. The discriminator is presence of `shape` + `dtype` + `layout` in the parsed JSON; iterating purely on `.json` extension would silently include transcripts and corrupt validate.py's tensor list. This catalog is the contract Stage 4 consumes to know exactly which tensors to emit and compare. ### Step 6: Provisional tolerances from per-tensor magnitude (execute) Write `tests/tolerances/.json` with one entry per tensor, derived from the magnitude statistics already recorded in each sidecar (`p99_abs` and `rms`): ``` max_abs = max(1e-4 × p99_abs, 1e-6) # 0.01% of typical large value mean_abs = max(1e-5 × rms, 1e-6) # 0.001% of RMS energy ``` When the same tensor name appears in multiple `(case, stage_dir)` sidecars, take the **max** p99_abs and rms across them so the budget covers the worst-magnitude instance. Per-tensor entry: `{max_abs, mean_abs, _provisional: true}`. Top-of-file `_comment` (array of strings) states: - These tolerances are magnitude-aware: `1e-4 × p99_abs` for max, `1e-5 × rms` for mean, with a `1e-6` floor. - They are provisional — Stage 4 finalizes them against observed C++ drift. - Entries with `_provisional: true` make finalization-not-yet-run obvious. - Do NOT ship a model while `_provisional` entries remain. Tensors well above this budget in Stage 4 are signals to investigate. ### Step 7: Reference WER baseline (execute) The Oracle WER run is the downstream accuracy baseline. Do not gate against publisher scores: Stage 4 and Stage 7 compare C++ against this measured reference run on our manifest and normalization. If intake captured a publisher score, compare and report the delta for context only. Keep every per-utterance reference hypothesis. Later WER drift should diff C++ vs captured reference `hyp_text` by `id` instead of re-running the reference. The acceptance dataset is `intake.upstream_benchmarks[0].dataset` (default LibriSpeech test-clean). Build the WER manifest once if it does not exist, then run the per-family reference runner over the **full** acceptance manifest and score it: ```bash DATASET=$(uv run python -c "import json; \ print(json.load(open('reports/porting///intake.json'))['upstream_benchmarks'][0]['dataset'].replace(' ','-').lower())") # Build the manifest if this dataset has not been ingested yet. [ -f samples/wer/${DATASET}.manifest.jsonl ] || \ uv run scripts/wer/ingest.py librispeech # or: fleurs --lang uv run --project scripts/envs/ \ scripts/wer/run_reference__.py \ --manifest samples/wer/${DATASET}.manifest.jsonl \ --model \ --out reports/wer/-REF.${DATASET}.jsonl uv run scripts/wer/score.py reports/wer/-REF.${DATASET}.jsonl # writes reports/wer/-REF.${DATASET}.score.json ``` This is a **one-time** reference run. Stage 4 and Stage 7 gate C++ WER against this same file; the reference is not re-run downstream. **GPU option (Modal).** To run the reference on cloud GPUs instead of locally, use the remote sweep — it installs the family venv on a GPU container and runs the same runner, returning the JSONL to score locally: ```bash modal run scripts/wer/remote/modal_sweep.py::reference_sweep \ --variants : --gpu L4 ``` It writes the same `reports/wer/-REF..jsonl`. See `scripts/wer/remote/REFERENCE_SWEEP_SPEC.md`. A new family needs a `scripts/wer/run_reference__*.py` with the uniform `--manifest/--model/--out/--device/--batch-size` contract (copy the closest same-framework one). ### Step 8: Sign-off Report: - Number of manifest cases dumped. - Total tensor count in `dump_coverage.json`. - Path to the manifest. - Tolerance summary: median and max `max_abs` across tensors so the user can eyeball whether the magnitude-aware numbers look sane (e.g. a logits-scale tensor should have a budget on the order of 1e-2; an embedding-scale tensor 1e-4). - Per-case transcript paths so the user can sanity-check the reference output before C++ work. - Reference WER baseline: measured reference WER on the acceptance dataset, the published score if present, the delta between them, the dataset + utterance count, and the `reports/wer/-REF..{jsonl,score.json}` paths. - Confirmation that the user reviewed the provisional tolerances. **Do not commit.** ## Postconditions - Every path listed in "Oracle packet contents" exists and uses the same `///` conventions. - Tensor sidecars include `rms` and `p99_abs`; `dump_coverage.json` catalogs tensors only. - `tests/tolerances/.json` is provisional, derived from per-tensor magnitude, every tensor carries `_provisional: true`, and the user reviewed it. - The measured reference WER baseline exists for the acceptance dataset, and any measured-vs-published gap was reported to the user. ## Pointers (read, not execute) - `docs/porting/2-artifacts-and-goldens.md` — manifest contract - `docs/porting/4-numerical-validation.md` — where this file's outputs flow next - `scripts/lib/ref_dump.py` — shared `write_tensor` / `write_transcript` - Existing dumpers as template references: - `scripts/dump_reference_parakeet_nemo.py` - `scripts/dump_reference_cohere_transformers.py` - `scripts/dump_reference_qwen3_asr_author.py` - `scripts/dump_reference_whisper_transformers.py` - Reference WER (Step 7): - `scripts/wer/run_reference__.py` — per-family reference WER runners (copy the closest same-framework one for a new family); local, GPU-accelerated if torch sees one - `scripts/wer/ingest.py` — builds the acceptance manifest (librispeech / fleurs) - `scripts/wer/score.py` — shared scorer; writes `.score.json`