Espdl Quantize

ESP-DL Quantization Tuning Skill

This skill turns the human "stare at error report, guess setting, rerun" loop into a structured, distribution-aware search. The user owns data loading and evaluation; the skill owns QuantizationSettingFactory.espdl_setting() and the iteration loop.

About <SKILL_DIR> in shell snippets below. This skill is agent-directory agnostic — it may be installed as .cursor/skills/espdl-quantize/, .opencode/skills/espdl-quantize/, or under any other agent's skills folder. Whenever you see <SKILL_DIR> in a shell command, substitute the absolute path of the directory containing this SKILL.md (the agent runtime gives you that path when it loads the skill). Setting it once at the start of a session makes the rest copy-pasteable:

SKILL_DIR=/abs/path/to/espdl-quantize   # the directory holding this SKILL.md

All in-skill markdown links (e.g. [scripts/run_iteration.py](scripts/run_iteration.py)) are already relative to <SKILL_DIR> and need no substitution.

Generality boundary

This skill is a general framework for any esp-ppq-supported graph: ResNet, EfficientNet, ViT, DETR, YOLO family, LSTM, custom CV/NLP backbones — anything espdl_quantize_torch / espdl_quantize_onnx can ingest. The Phase 2 calib×TQT cartesian product, the Phase 3 lever ordering, and the four Composition discipline rules are all distribution-driven; none of them depends on a specific network structure or family. The Worked example at the end uses MobileNet-V2 on ESP32-P4 as an empirical demonstration — its concrete numbers are illustrative, not a model-selection threshold.

Why this skill exists

esp-ppq exposes a dozen tunable passes (calibration algorithm, layerwise equalization, bias correction, weight split, mixed precision via dispatching table, TQT, LSQ, blockwise reconstruction, ...). Each has 2-6 parameters. Trying them by hand is slow and error-prone.

What this skill brings to the table:

1. Knowledge — every esp-ppq method's principle, parameters, applicable scenarios, and anti-patterns are codified in references/ppq_methods.md.
2. A decision rulebook — given the top-K worst layers' input/weight/output distributions, references/decision_playbook.md maps observed patterns to candidate methods.
3. A fixed harness — scripts/run_iteration.py takes the user's contract module plus a JSON setting and emits structured artifacts (metrics, layerwise error, per-layer stats) so the agent only has to read JSON to make the next decision.
4. A search state machine — scripts/compare_iterations.py inspects what's already been tried and tells the agent which iteration to run next via comparison.json["next_step_hint"]. The hint embeds a complete setting.json template so the agent only has to fill in the rationale.
5. Target-aware safety net — the harness detects passes that conflict with the target's quantization policy:
   • LSQ on POWER_OF_2 targets (esp32p4 / esp32s3 / c) — auto-disabled. esp-ppq's LSQDelegator silently disables continuous-scale training under POWER_OF_2, so the pass would degenerate to weight-only tuning while paying full TQT-level PC time. Use TQT instead — it trains log2_scale and is POWER_OF_2-native.
   • Layer-wise equalization on esp32p4 — warn-only (changed in this revision). esp-ppq officially marks the combination as "Not recommend" (see esp-ppq/md_doc/Passes/LayerwiseEqualization.md, "Usage" section), but empirical runs show some MobileNet-family / depthwise-separable networks still benefit. The harness now lets the pass run when equalization.enabled=true and emits a strong warning; the agent should treat it as a Phase 3 lever to try only after the calib×TQT cartesian product has settled.

What the user has to provide

A single Python module (typically named user_quant.py) that exports:

• QUANT_CONFIG dict — model path, input shape, target chip, bits, primary_metric, etc.
• create_calib_dataloader() — returns the calibration DataLoader.
• evaluate(quant_graph) — returns a dict whose keys include QUANT_CONFIG["primary_metric"].
• For torch flow only: get_torch_model() — returns the nn.Module.
• Optional: collate_fn(batch) and evaluate_fast(quant_graph).

The full contract spec is in references/contract.md. Two ready-to-copy examples live in assets/user_quant_torch_example.py and assets/user_quant_onnx_example.py.

The skill never edits the contract module. All iteration state lives under outputs/.

High-level flow

flowchart TD
    contract[user_quant.py] --> harness[run_iteration.py]
    setjson[outputs/iter_N/setting.json<br/>written by agent] --> harness
    harness --> ppqapi["esp_ppq.api.espdl_quantize_torch / _onnx"]
    ppqapi --> graph["esp-ppq BaseGraph<br/>(esp_ppq.IR.BaseGraph)"]
    graph --> lwerr["layerwise_error_analyse"]
    graph --> stat["statistical_analyse"]
    graph --> evalfn[user.evaluate]
    lwerr --> art[outputs/iter_N/]
    stat --> art
    evalfn --> art
    art --> compare[compare_iterations.py]
    compare --> hint["comparison.json<br/>next_step_hint"]
    hint --> phase{phase?}
    phase -- "1 / 2 / 3" --> agentTpl["agent fills rationale on embedded template"]
    phase -- "5 (open exploration)" --> agentFree["agent reads phase5_signals + artifacts,<br/>writes setting.json from scratch"]
    phase -- "4 (final)" --> finalize["outputs/best/ + outputs/final_report.md"]
    agentTpl -.writes next setting.-> setjson
    agentFree -.writes next setting.-> setjson

In Phases 1-3 the agent's job each round shrinks to: read comparison.json["next_step_hint"], copy the embedded setting.json template, fill in the rationale field, run the harness. In Phase 5 the script stops prescribing settings — the agent reads phase5_signals plus the per-iteration error artifacts and writes the next setting.json from scratch (see "Phase 5 — Agent-driven exploration" below).

Phases

Phase 0 — Validate contract and environment

Important — ignore Docker / image / /work mentions you may see elsewhere. Some user projects (and user_quant.py itself) still carry comments left over from an older Docker-based workflow — phrases like "build the image", "Phase 0 — docker 准备", /work inside Docker, or docker run --gpus all. Those are legacy text only, not steps to execute. The skill now runs entirely in the current Python interpreter.

Before any quantization, do these once per session:

1. Make sure the current Python environment has esp_ppq (with the [cpu] extra), torch, plus the small set of helpers the harness needs:

   pip install -e <path/to/esp-ppq>[cpu]
   pip install -r "$SKILL_DIR/assets/extra_requirements.txt"
   

   The skill is environment-agnostic — it does not require Docker. As long as python -c "import esp_ppq, torch, onnx, onnxsim, pandas, scipy, tqdm" succeeds, you are ready to go.

2. Validate the user's contract module imports cleanly and exposes the required functions/keys:

   python "$SKILL_DIR/scripts/run_iteration.py" \
     --user-quant <path/to/user_quant.py> \
     --output-dir <path/to/user_project>/outputs/contract_check \
     --check-contract
   

3. Make sure the iteration workdir exists (default: <contract_dir>/outputs/). The harness creates it on first run.

The working directory for python should be the directory containing user_quant.py (or any directory — the harness resolves relative paths in QUANT_CONFIG against the contract module's directory).

Phase 1 — Baseline (iter_0)

Run the default QuantizationSettingFactory.espdl_setting() once. The agent does NOT propose any settings here — the harness uses a built-in baseline JSON when --baseline is passed.

python "$SKILL_DIR/scripts/run_iteration.py" \
  --user-quant <path/to/user_quant.py> \
  --output-dir <path/to/user_project>/outputs/iter_0 \
  --baseline

After it finishes, read these files:

• outputs/iter_0/metrics.json — what evaluate() returned, plus _primary shortcut.
• outputs/iter_0/layerwise_error.json — {op_name: snr} sorted descending by error. Covers only is_computing_op (Conv / Gemm / ConvTranspose / MatMul / Attention / PPQBiasFusedMatMul / LSTM); the SNR is the isolated contribution of that op when it alone is quantized.
• outputs/iter_0/layer_stats.json — statistical_analyse filtered by the layerwise top-K (legacy artifact; same coverage as layerwise).
• outputs/iter_0/layer_stats_full.json — (new) the full statistical_analyse output: every non-passive op's per-input/per-output distribution + cumulative SNR. This is the only artifact that includes Add / Concat / Resize / AveragePool / Sigmoid / Softmax / GRU / LayerNorm.
• outputs/iter_0/non_computing_hot_ops.json — (new) the top-K non-COMPUTING_OP layers ranked by max per-variable SNR, plus inputs_float_std_ratio (max/min Float Std of input variables, used by playbook rule R8).
• outputs/iter_0/graphwise_jumps.json — (new) adjacent computing-op pairs whose cumulative SNR gap is not explained by the downstream op's isolated contribution. Lists the intervening non-computing ops as suspected culprits.
• outputs/iter_0/console.log — full stdout/stderr.

Tell the user the baseline numbers, the top-5 error layer names from layerwise, and (if non-empty) the top-3 entries from non_computing_hot_ops.json. The state machine in compare_iterations.py decides when to finalize — do not stop here on your own even if iter_0 looks like it hit target_metric; run the comparison once and let next_step_hint["phase"] == "phase-4-final-report" confirm.

Phase 2 — Calibration × TQT(default) cartesian product (mandatory)

This phase must run three iterations in strict sequence, each enabling exactly two fields: calib_algorithm and tqt_optimization (with the esp-ppq default schedule). No other pass is enabled. The cartesian product is what makes the search robust — calibration in esp-dl quantization is not separable from the training pass: a calibration that regresses standalone may become the strongest base when paired with TQT, and vice versa. See the Worked example below for the empirical case that motivated this design.

The TQT default schedule is strict:

{
  "lr": 1e-5,
  "steps": 500,
  "block_size": 4,
  "is_scale_trainable": true,
  "gamma": 0.0,
  "int_lambda": 0.0,
  "collecting_device": "cuda"
}

Iteration sequence:

The way to drive this is to run scripts/compare_iterations.py between iterations:

python "$SKILL_DIR/scripts/compare_iterations.py" \
  --output-dir <path/to/user_project>/outputs

comparison.json["next_step_hint"] will be phase-2-calib-tqt-sweep until all three calibrations are covered with TQT(default), and the embedded setting.json template can be copied verbatim into outputs/iter_<N>/setting.json (only fill in the rationale).

Critical: iterations are strictly sequential — never run two in parallel. Single GPU, calibration-data download race, and any of the three legs can short-circuit the rest if it hits target_metric. If you spawn parallel subagents the search breaks.

Phase 3 — Residual fixes from best-so-far

After Phase 2 the best-so-far iteration is the one with the highest primary_value among iter_0..3. comparison.json["next_step_hint"] switches to phase-3-residual (or phase-3-pivot if the last two iterations both regressed vs best).

Each Phase 3 iteration must mutate from comparison.json["best_iteration"]'s setting.json and change exactly one thing. The lever order below is the linear default for deploy_runtime_priority="balanced"; lever 3a-3 is conditional (entered only when one of two specific signals fires); the speed-priority reorder is described under Accuracy-first method ordering below.

The state machine in compare_iterations.py automatically picks the next lever per the table above and the deploy_runtime_priority knob. The agent's job each Phase-3 iteration shrinks to: read comparison.json["next_step_hint"]["advice"], copy the embedded change snippet onto best-so-far's setting.json, fill rationale with the specific layer-stats observation that drove the choice, run the harness.

Stop conditions are no longer the agent's call — when any of the following hold, the state machine yields. Two of them finalise (Phase 4); two of them hand control to the agent (Phase 5). Note that the Phase-3 cap fires at 5 iterations even though the linear-order list has 8 levers — see "Why _PHASE3_CAP=5 leaves untried linear-order levers" in the Phase 5 section below for the trade-off and how Phase 5 picks up the slack.

Lever 3c is correctly skipped when R8 doesn't fire on best-so-far's non_computing_hot_ops.json (see "R8 trigger" below): the data says fusion alignment would regress, so the state machine doesn't burn an iteration on it and instead advances to 3d. The skip is recorded in comparison.json["next_step_hint"] so the agent (and the human reviewing later) sees why the lever was bypassed.

Phase 5 — Agent-driven exploration

Phase 5 exists because the Phase-3 linear search is, by design, surgically narrow. Each Phase-3 lever changes exactly one thing on top of best-so-far. That's the right shape when you're hunting for the next single biggest fix, but it can't find combinations: a recipe that needs `percentile + TQT + equalization + 3-layer int16

• bias_correct(theexample_quantize_mobilenetv2_esp32p4/outputs/winning configuration, iter_14) is unreachable from any single Phase-3 lever applied to any single Phase-2 leg. Worse, Composition discipline #4 (calibration is not separable from the training pass) means even Phase 2 cannot tell you whetherpercentile` becomes the best calibration once equalization + int16 + bias are stacked on top of it.

In Phase 5 the state machine yields and the agent drives. The contract:

• The hint in comparison.json["next_step_hint"]["advice"] is meta-guidance, not a setting.json template. There is no iteration_id skeleton to fill in — you write the next setting.json from scratch.
• The hint is paired with a structured comparison.json["next_step_hint"]["phase5_signals"] block that summarises history: which iterations improved over their prior best (and by how much), which regressed (so you don't re-stack their changes), what calibration algorithms haven't been tried on top of the current lever stack, and pointers to the on-disk artifacts to consult before proposing the next change.
• Each Phase-5 iteration must still mutate from best-so-far (Composition discipline #1) and stop escalating after one regression (#3). Discipline #2 (one knob per iteration) is relaxed — see Composition discipline #5 below.

Inspiration patterns (these are starting points; let the actual data decide which one applies on your model):

• STACK improving levers. When phase5_signals.improving_levers lists ≥2 entries (e.g. iter_5 enabled tqt_optimization, iter_9 enabled equalization), the first natural Phase-5 iteration is one that turns BOTH on at once. The mobilenetv2-p4 path went iter_11 (TQT + equalization + int16x3) → iter_13 (added calibration swap, +0.55%) → iter_14 (added bias_correct, +0.05%, final best).
• CROSS-POLLINATE CALIB. When phase5_signals.untried_calib_swaps is non-empty, run a single iteration that swaps the calibration on top of the current lever stack. The Phase-2 cartesian product evaluates calib × TQT(default) in isolation; once 3-4 levers are stacked the ranking can flip — exactly what happened on mobilenetv2-p4 when percentile, which had previously been considered a calibration loser, became the winner once stacked with TQT + equalization + int16.
• ABLATE. Once Phase 5 finds a new best, drop one component at a time and check whether accuracy stays above target. This produces a "minimal recipe": fewer passes, fewer surprises, shorter PC quantization time. Two ablation directions are particularly useful — drop the highest-cost component (e.g. one int16 op, or weight_split) for on-device speed; drop a Tier-A pass (TQT off, equalization off) to test which passes are actually load-bearing.
• DIVE INTO ARTIFACTS. When the above three don't produce an obvious next move, open best's layerwise_error.json, layer_stats_full.json, non_computing_hot_ops.json, and graphwise_jumps.json. Pick a layer with a concrete distribution observation (e.g. "Conv layer X has Float Std skew >5 but the weight per-channel max/mean is only 2.3 — equalization won't help; this is a high-variance activation, try TQT escalation or int16 on this single op") and write the next iteration around that observation. Cite the file + the number in rationale.

Stop signals (each → finalize). Two are auto, two are agent-driven:

1. primary_metric reached target_metric — compare_iterations.py AUTO-finalizes via phase-4-final-report. final_report.md records Stop reason category: target_reached.

2. Plateau — last 3 iterations all within 0.1% relative of best. compare_iterations.py AUTO-finalizes via phase-4-final-report. final_report.md records Stop reason category: plateau plus the three plateau values.

3. User-given iteration budget reached — agent runs --finalize --force-finalize NOW, regardless of phase and regardless of remaining untried patterns/levers. User budget is the hard ceiling. --force-finalize is the explicit opt-in that confirms "this early stop is intentional"; final_report.md records Stop reason category: force_finalize_phase5 plus the untried lists so the user can see what was skipped.

4. Coverage-exhausted "ran out of ideas" — STRICT. Only fires when ALL the following hold simultaneously:

   • the user did NOT give a specific iteration budget;
   • phase5_signals.untried_phase5_patterns is empty (every pattern attempted at least once);
   • phase5_signals.untried_phase3_levers is empty (every linear-order Phase-3 lever either tried or correctly skipped by its trigger);
   • phase5_signals.untried_5beta_reapply is empty (every calib swap re-tested on the current deepest stack — see "5β-reapply" below);
   • the most recent iterations did not produce a new best.

   If signal (3) is in play, signal (4) is disabled. Keep iterating until the user budget is met, drawing fresh variations from the untried lists. Phase 5 has NO hard iteration cap from the state machine; the user is the cap.

The comparison.json["early_finalize_command"] field always contains the one-line --finalize invocation; the stdout "Tip" block reprints it after every run.

Hard-reject of premature --finalize: if you run --finalize while still in phase-5-agent-driven and neither target nor plateau is met AND you do NOT pass --force-finalize, compare_iterations.py PRINTS THE REJECTION BLOCK, REFUSES TO WRITE outputs/best/ or outputs/final_report.md, and EXITS WITH CODE 1. The agent must either (a) re-run without --finalize and keep iterating, or (b) pass --force-finalize to confirm intentional early stop. This is the operational enforcement of the user-budget contract in signal (3); see "How premature finalize is prevented" below for the rationale.

5β-reapply (the high-leverage coverage gap that needs explicit tracking): Composition discipline #4 says calib-only ranking does not predict the combined ranking, but Phase 2 runs before any levers are on. The corollary is that an early 5β CROSS-POLLINATE attempt on a shallow stack can produce a misleading verdict — the same calib swap on the current deepest stack may behave very differently. The canonical example is example_quantize_mobilenetv2_esp32p4 iter_13: percentile lost to kl on the Phase-2 stack but won by +0.55% on the deepest lever stack. The skill tracks this via phase5_signals.untried_5beta_reapply: a list of calibrations that appeared as 5β targets earlier in history but were not re-tested on the current best's stack. The Phase 5 hint surfaces the list explicitly, and stop signal (4) is blocked while it is non-empty.

Tunable-params soft advisory: the Phase 5 hint includes a "Tunable parameters in current best" section listing the parameter knobs available inside each enabled pass (TQT lr / steps / block_size, blockwise lr / steps / block_size, equalization opt_level / iterations, fusion_alignment direction, percentile calibration percentile) with common value ranges drawn from references/ppq_methods.md. This is a SOFT advisory — NOT part of coverage. The agent reads the section, decides whether the layerwise / non_computing_hot_ops data justifies a knob change, and proposes the next iteration accordingly. Tuning a parameter within an already-enabled pass is a valid Phase-5 move; not every variation requires turning a pass on/off.

User-budget enforcement in Phase 5

Phase 5's "no hard cap" property means the state machine will keep emitting hints forever if you let it. The user budget is what bounds the loop. Concretely:

• Track the user-budget count in your head (or in scratchpad). Increment after each iteration completes.
• After the N-th iteration where N == user budget, run the --finalize command and stop.
• Never invoke stop signal (4) when a user budget is in play — even if all patterns and levers are covered. Spend the remaining budget on variations of attempted patterns: re-stack improving levers in different combinations, try the same pattern on a different layer subset, ablate a different component, dive into a different artifact than last time. Variation under user budget is REQUIRED; fabricating defensible variations is part of the Phase 5 contract.
• The mechanism that prevents the wrong call here is rationale citation discipline #5: every iteration must name the iter id(s) whose data motivated the change. If you cannot name any prior iter that motivates the next change AND the user budget remains, look at a different artifact, find a number you can name, and use that as your rationale — do not finalize.

Why _PHASE3_CAP=5 leaves untried linear-order levers (by design)

The Phase-3 linear-order lever list has 8 entries (3a-1, 3a-2, 3b, 3c, 3d, 3e, 3f, 3g), but _PHASE3_CAP=5 caps the number of structured single-knob Phase-3 iterations at 5. The trade-off this cap encodes:

• Pro: Phase 5 can start exploring sooner (cross-pollination + ablation + stacking are higher-leverage moves than the tail of the linear list in many real cases — see the mobilenetv2-p4 iter_13 +0.55% jump).
• Pro: when target_metric is hit early in Phase 3, the cap is irrelevant — the short-circuit fires first.
• Con: levers near the end of the linear order (typically 3d / 3e / 3f / 3g) are often left untried when the cap fires. The 3a-3 conditional path can also occupy a cap slot, making this worse.

The skill closes the gap by treating those untried levers as first-class Phase 5 coverage targets. phase5_signals.untried_phase3_levers lists them by id; the hint explicitly directs the agent to STACK each onto best-so-far as a Phase-5 iteration before stop signal (4) can fire. Functionally a Phase-3 single-knob mutation and a Phase-5 STACK iteration produce the same setting (both mutate from best + flip one lever), so coverage is preserved — just under a different label and with a slightly larger lever-stack baseline.

Worked example: in example_quantize_mobilenetv2_bad_esp32s3/outputs/, the 20-iteration run only produced 12 iterations because the agent (a) interpreted the old "Pretending there's a 5th idea when there isn't one is worse than finalising" line as permission to bail, and (b) had no visibility into the fact that 3f weight_split and 3g blockwise_reconstruction were untouched and 5gamma ABLATE + 5delta DIVE-INTO- ARTIFACTS were untried Phase-5 patterns. Under the current contract:

• The "Pretending..." sentence is gone — replaced by the hard rule that user budget trumps signal (4).
• The hint now surfaces untried_phase3_levers=[3f, 3g] and untried_phase5_patterns=[5gamma, 5delta] as named targets.
• Premature --finalize in phase-5 (target not met, no plateau) is HARD REJECTED: compare_iterations.py refuses to write outputs/best/ / outputs/final_report.md and exits with code 1. The agent must either continue iterating or explicitly pass --force-finalize. The earlier soft-warning version of this guardrail was a load- bearing failure mode that produced both this 12-of-20 run and the later 18-of-30 / 21-of-30 runs in example_quantize_mobilenetv2_esp32p4_tmp/outputs/.

These three together make the iteration-budget mismatch self-correcting.

How premature finalize is prevented

The hard-reject contract for --finalize is the operational enforcement of the user- budget rule in stop signal (3). Concretely:

1. The agent (or user) runs compare_iterations.py --finalize while phase == phase-5-agent-driven and the script computes target_metric is NOT reached and the recent iterations are NOT a plateau.
2. compare_iterations.py prints the rejection block, listing untried_phase5_patterns, untried_phase3_levers, and untried_5beta_reapply.
3. comparison.json is still written (so the agent can re-read the hint), but outputs/best/ and outputs/final_report.md are NOT touched.
4. The script exits with code 1.
5. The agent picks ONE of:
   • re-run without --finalize and continue iterating (use the printed untried lists as concrete next targets); OR
   • pass --force-finalize alongside --finalize to confirm intentional early stop. The resulting final_report.md has a ## Stop reason section with category force_finalize_phase5 plus the untried lists so the user can see exactly what was skipped.

Why hard-reject (not soft warning): the soft warning was ignored both in the bad- esp32s3 12-of-20 run and in the esp32p4_tmp 18-of-30 / 21-of-30 runs. Making the reject load-bearing means the agent literally cannot produce a final_report.md by accident in phase-5 — a positive write requires --force-finalize, which the agent will only emit when the user budget rationale is solid.

Phase 4 — Final report

Two ways to enter Phase 4:

• State-machine trigger (machine view): comparison.json["next_step_hint"]["phase"] == "phase-4-final-report". The state machine emits this when target reached, plateau, Phase-3 cap, or all linear-order levers tried (see Phase 3's stop-condition list).
• User-budget trigger (human view): the user gave the agent a specific iteration budget — phrasings like "iterate 3 times", "迭代 3 轮", "只跑 N 轮", "iterate N times", "最多 N 轮", "only N iterations". When this budget is hit, the user-budget trigger always wins even if the state machine still wants to keep going.

Why the auto-finalize is bullet-proof. compare_iterations.py writes outputs/best/ and outputs/final_report.md whenever either trigger fires:

• Automatic when phase == "phase-4-final-report" — every invocation of compare_iterations.py --output-dir <outputs> checks this and finalizes if true. Agents reading the script's stdout will see a [compare] phase-4 detected; finalize results: ... block.
• On demand via the --finalize flag at any time, regardless of phase. This is the escape hatch for the user-budget case — agents should copy the command from comparison.json["early_finalize_command"] (or the printed "Tip: how to wrap up at any time" block at the bottom of compare_iterations.py's stdout) and run it after the last user-budgeted iteration completes.

The generated final_report.md carries an HTML marker comment on its first line. Subsequent finalize runs detect the marker → safely refresh the report (no data loss). If an agent has hand-edited the file (and removed the marker), subsequent finalize preserves it untouched unless --force is passed. Sections ## Key findings and ## Remaining gap (if target not met) are seeded with auto-bullets but agents are explicitly invited (via the marker comment) to expand them with concrete distribution interpretations from layer_stats.json / non_computing_hot_ops.json / graphwise_jumps.json.

Iteration history table — new columns. The auto-generated table now includes:

• rank — dense ranking by primary_value (1 = best). Recomputed from disk on every finalize, so adding more iterations later won't reverse the relative order of any pre-existing pair (this is asserted by the unit tests). Columns visible in the report and in comparison.json["iteration_ranks"].
• affects inference speed — "No" for almost all settings; "Yes (...)" only when the iteration enables dispatching_table int16 promotion or weight_split (the only two passes with permanent on-device runtime cost; see "On-device runtime cost cheat-sheet"). When the best iteration has affects inference speed = Yes, inspect the rank-2/3 rows — if they trade < 0.1% accuracy for affects inference speed = No, the user may prefer the runner-up for production deployment.

Recommended agent workflow after finalize:

1. Read outputs/final_report.md. The Summary, Iteration history (with rank + speed columns), Best setting, Python snippet are auto-generated; expand ## Key findings and ## Remaining gap with concrete bullets if the model warrants.
2. Run a single full-eval re-check to replace the iteration loop's evaluate_fast() number with the user's real evaluate(): python {SKILL_DIR}/scripts/run_iteration.py --user-quant <...> --setting outputs/best/setting.json --output-dir outputs/iter_<NEW> --use-full-eval. If the resulting <primary_metric> differs from what's in the Summary, update the Summary line in final_report.md. The marker line keeps the file refreshable; once you remove the marker (or pass --force from the script), subsequent automated runs won't clobber edits.
3. If you ever need to regenerate the report from scratch (e.g. after fixing a bug in an iteration), run: python {SKILL_DIR}/scripts/compare_iterations.py --output-dir <outputs> --finalize --force.

Legacy fallback. The pre-auto-finalize workflow (manually run --write-best, hand-write outputs/final_report.md) is still supported for completeness — if for any reason compare_iterations.py does not emit the artifacts (e.g. broken iteration data on disk), the agent can fall back to that flow. The --write-best flag now writes only outputs/best/; the report remains the agent's responsibility in that fallback.

Final-report template (auto-emitted by the script, for reference / audit):

<!-- auto-generated marker line — agents may edit Key findings / Remaining gap -->
# Final Report: <model> on <target>

## Summary
- Best iteration: iter_<N>
- <primary_metric>: <value> (target_metric=<target or "not set">)
- _Note: value comes from evaluate_fast(); run --use-full-eval to refresh._
- On-device speed cost vs baseline (best): <No | Yes (...)>
- Other metrics: <copy from outputs/best/metrics.json>

## Iteration history
| iter | method changed | <primary_metric> | delta | outcome | rank | affects inference speed |

## Best setting
<inline the FULL outputs/best/setting.json>

## Python snippet
<auto-translated QuantizationSettingFactory.espdl_setting() recipe>

## Key findings
<auto-bullets — agents extend with concrete distribution interpretations>

## Remaining gap (if target not met)
<auto-bullets — agents replace boilerplate with model-specific recommendations>

Composition discipline (read before every iteration)

These rules govern the iteration loop. Violating any of them = current iteration is discarded, agent rolls back to best-so-far and re-runs.

1. Mutate from best-so-far, not the last iteration. Always start the next setting.json from comparison.json["best_iteration"]["dir"]/setting.json. The most recent run can be a regression you should not inherit.

2. One new method (or one parameter change) per iteration (Phases 1-3 only). Calibration algorithm and tqt_optimization with the default schedule are treated as the conjoined Phase 2 base — they enter and leave together inside Phase 2. Outside Phase 2 and inside Phase 3, change exactly one knob. If two changes are stacked and the iteration regresses, you can't tell which one hurt. (Phase 5 relaxes this rule — see #5.)

3. Stop escalating after one regression. If iter_N raises a TQT hyper-parameter (or tightens a lever) and the metric drops, do not push further on that axis; pivot to a different lever from the Phase 3 table.

4. Never retire a calibration algorithm based on its calib-only score. Calibration is the input distribution shaper for downstream passes (especially TQT in esp-dl, where POWER_OF_2 makes TQT the only available training-based pass). Calib-only accuracy does not predict combined accuracy: percentile may regress standalone but become the strongest TQT base, because tail-clipping leaves more "training space" for TQT to recover. Phase 2 must always evaluate calibration with calib × TQT(default) cartesian product, never with calib-only ranking. The same principle applies in Phase 5: calibration ranked against the Phase-3 lever stack can flip vs the Phase-2 cartesian ranking — re-test untried calibrations after the lever stack settles. See Worked example below for the concrete reproducer.

5. Phase 5 multi-knob changes are allowed iff the rationale cites historical evidence. The "one knob per iteration" rule (#2) is relaxed in Phase 5 — an iteration may stack 2+ passes — but only when each pass change names the specific iteration whose data motivates it (e.g. "iter_5 showed lever 3a-3 stabilised the perturbed layer; iter_9 showed equalization improved bottleneck Conv chains; combining them tests whether the gains compound"). Without citations, a multi-knob change is a guess and must be split into single-knob st

…(truncated)