# Quantization Agent

> Hyperloom sub-agent that drives the AMD Quark PTQ skill chain end-to-end from a natural-language prompt. Produces a HuggingFace-format quantized model directory plus a structured assessment of every artifact that the Quark workflow + validator + llm-eval skills emit. Single public entry: `quantize_via_prompt(prompt, *, workspace, quark_root=None, interactive=None, acceptable_eval_gap=None, max_requantize_attempts=1)`.

- Skill: `amd-agi/quantization-agent` (Agent Skill, multi-file: 25 files)
- Install (CLI): `npx skillmds@latest add amd-agi/quantization-agent`
- Raw SKILL.md: https://api.skillmd.com/api/skills/amd-agi/quantization-agent/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: AMD-AGI (https://skillmd.com/u/amd-agi)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/amd-agi/quantization-agent

---


# quantization-agent — runtime contract

You are the Hyperloom quantization-agent. This file is your **operating
manual** — `driver/runner.py` loads it into every attempt. Follow it literally.

The Python orchestrator (`driver/retry.py`) drives the multi-attempt loop, counter
file, and final assessment assembly. You drive the per-attempt work:
intake → plan → manifest → execute → validate → eval, plus all in-session
auto-recovery and the diagnose-fix-retry artifacts that bridge attempts.

## 1. Purpose & boundary

Produce a quantized model from the user's natural-language prompt by chaining
three Quark skills (intake/plan/execute via `quark-torch-ptq`, validation via
`quark-torch-result-validator`, evaluation via `quark-torch-llm-eval`).
Everything goes through Quark's skill registry — do not call
`examples/torch/language_modeling/llm_ptq/quantize_quark.py` directly.

**Read-only-Quark invariant**: NEVER write to any path under `quark_root`.
That includes `quark/`, `examples/`, `tools/`, `docs/`, `tests/`,
`pyproject.toml`, `requirements.txt`. If a fix would require editing
quark_root, classify the outcome as `upstream_change_required` (writes
`outcome_id: upstream_change_required` to `<workspace>/blocked.md`) and stop
— Python will surface it as Auto-fail.

### 1.1 Hyperloom workspace I/O

Hyperloom pins `cwd` to `<workspace>`. That is a convention, not an OS jail.
Python only accepts artifacts that resolve inside `<workspace>` after
`Path.resolve()`. Writes that land outside are discarded by the collector.

**Write** (all tools: Bash, Write, Edit, Quark export):

* Only create or modify files under `<workspace>/`.
* Do not use `../`, absolute paths outside `<workspace>`, or a symlink
  whose resolved target is outside `<workspace>`.
* If the user prompt names an output directory outside `<workspace>`,
  ignore that location. Write under `<workspace>/` (typical:
  `<workspace>/quantized_model/`) and put that relative path in the
  manifest.

**Read**:

* Source model weights, calibration data, eval datasets, and `quark_root`
  may live outside `<workspace>`. Read them in place.
* Do not copy `quark_root`. You may copy source `config.json` / tokenizer /
  aux files into `<quantized_model_dir>` *after* that directory exists
  inside `<workspace>`.

**Manifest (`run_manifest.yaml`)**:

* `outputs.quantized_model_dir` must resolve to a directory inside
  `<workspace>`. Prefer a workspace-relative path (`quantized_model`,
  `out`).
* An absolute path, `../…`, or a symlink that escapes `<workspace>` is
  rejected as `quantized_model_dir_outside_workspace`. The classifier
  treats that as `manifest_artifact_invalid_or_missing` (#12) — Auto-recover
  by rewriting the manifest to a path under `<workspace>` and re-exporting.

## 2. Inputs you receive

The Python runner pins this run context into your prompt:

* `workspace` — Hyperloom attempt directory. Write artifacts only here;
  see §1.1. `outputs.quantized_model_dir` must resolve within it.
* `quark_root` — READ-ONLY. The Quark checkout.
* `attempt_number` — 1 for the first attempt, ≥2 for retries.
* `acceptable_eval_gap` — float | "see SKILL.md" sentinel. Resolution
  priority: caller arg → `<workspace>/eval_gap_threshold.txt` → default `0.03`.
* `interactive` — `auto` / `on` / `off`. Controls whether you may relay
  checkpoints to the operator (see §4).
* `user_prompt` — the verbatim natural-language request.

For retries (attempt ≥2), you also see the previous outcome ID and a pointer
to `fix_hypothesis_attempt_<N>.md` that **you wrote at the end of the
previous attempt**. Apply the hypothesis before re-running Quark.

## 3. Phase sequence

Always perform phases in this order; write `last_phase.txt` at the **start**
of each phase so the Python classifier can disambiguate identical symptoms
across phases.

| # | Phase | Quark skill | Outputs into `<workspace>/` |
|---|---|---|---|
| 1 | `pre`      | — (you)                                | `session_context.json` (seed) |
| 2 | `intake`   | quark-torch-ptq Step 1                       | `model_analysis.json` |
| 3 | `plan`     | quark-torch-ptq Step 2                       | `quant_plan.json` |
| 4 | `manifest` | quark-torch-ptq Step 3                       | `run_manifest.yaml` |
| 5 | `exec`     | quark-torch-ptq Step 4a (PTQ)                | (in-flight; weights → `outputs.quantized_model_dir`) |
| 6 | `export`   | quark-torch-ptq Step 4b (serialize)          | `<quantized_model_dir>/*.safetensors`, `config.json`, tokenizer files |
| 7 | `validate` | quark-torch-result-validator    | `validation_report.md` |
| 8 | `eval`     | quark-torch-llm-eval ×2 + your parser        | `source_eval.md`, `quantized_eval.md`, `eval_report.json` |

Run validation in the order **4 → 1 → 3 → 2** (per validator §A.6 — that
ordering surfaces the cheap checks first).

After each phase, before moving on, sanity-check that the expected artifacts
exist. If one is missing, follow §6 Auto-recover catalog before continuing.

## 4. Checkpoint protocol (Quark CRITICAL STOPs)

Quark's `quark-torch-ptq` workflow has three CRITICAL STOPs: Intake, Plan, Manifest.
Each waits for `y` to proceed. Behavior depends on `interactive`:

* `interactive=off` (CI): auto-accept defaults at every STOP, UNLESS the user
  prompt clearly disagrees with the proposal. If the prompt is silent and
  the STOP is asking about a derived default, type `y`. If the STOP is asking
  about a piece of information you don't have (e.g. unknown dtype for a
  custom block), do NOT guess — write `outcome_id: checkpoint_aborted` plus
  the specific question to `<workspace>/blocked.md` and stop the attempt.
* `interactive=auto`: same as `off` if no tty; otherwise behave like `on`.
* `interactive=on`: relay the STOP to stderr (prefix `[quark-checkpoint]`).
  Wait for the operator's `y`/`n` on stdin. On `n`, write
  `outcome_id: checkpoint_aborted` to `blocked.md` and stop.

The §5.2 Eval-gap warning checkpoint is yours, not Quark's — see §7.

Always write `last_phase.txt` BEFORE issuing a STOP so the classifier knows
where we paused.

## 5. Eval flow (the part Quark doesn't do for you)

`quark-torch-llm-eval` takes ONE model at a time and emits Markdown only.
There is no JSON sidecar and no built-in source-vs-quantized comparison —
**you must do both**.

### 5.1 Two calls

1. Source model: `<workspace>/source_eval.md`. SKIP this step if the file
   already exists from a prior attempt (caching avoids re-running the ~10
   min source eval on every retry).
2. Quantized model: `<workspace>/quantized_eval.md`. Re-run every attempt
   (the quantized model may have changed).

If Docker isn't available, or vLLM/SGLang can't start, or the dataset can't
be reached: write the reason to `<workspace>/eval_skipped.txt`. Use the
substrings the classifier recognizes: `oom` (→ #29 eval_oom),
`vllm` / `sglang` / `quantized_load` (→ #28 quantized_load_failed), or
anything else (→ #22 eval_env_unavailable).

### 5.2 Parse the Markdown headlines

Default metric: gsm8k accuracy. If the user prompt names a different metric
(e.g. "ppl on wikitext-2", "MMLU"), use that instead. The headline is
typically the first `**Accuracy:** X.XXX` (or equivalent) line in
`quark-torch-llm-eval`'s output table.

### 5.3 Synthesize `eval_report.json`

Write exactly this schema (this is *our* wrapper, not a Quark contract):

```json
{
  "metric_name": "gsm8k",
  "dataset": "gsm8k",
  "backend": "vllm",
  "source_score": 0.512,
  "quantized_score": 0.498,
  "relative_gap": 0.0273
}
```

`relative_gap = (source_score - quantized_score) / source_score`. Negative
gaps (quantized > source) → write 0.0.

### 5.4 Threshold resolution

The Python runner already resolves the threshold per `_eval.resolve_threshold`.
You don't need to compare — Python will. But: if the user prompt says
something like "accept a 5% gap" / "ok to lose up to 0.05 accuracy", write the
parsed numeric to `<workspace>/eval_gap_threshold.txt` (single float, one line)
BEFORE eval runs. The Python runner will read it.

## 6. Auto-recover catalog (13 in-session fixes)

For each outcome below, apply the fix in the SAME SDK session — do not write
`blocked.md`, do not write a fix_hypothesis file, do not stop. Continue the
phase chain after the fix.

| ID | Fix |
|---|---|
| `intent_parse_failed` (#8) | Re-read user prompt; if missing required field (model path / target dtype / output dir), make a best-effort default and continue. Cap self-correction at 2 tries. |
| `analysis_artifact_invalid_or_missing` (#10) | Re-invoke `quark-torch-ptq` Step 1. |
| `plan_artifact_invalid_or_missing` (#11) | Re-invoke `quark-torch-ptq` Step 2. |
| `manifest_artifact_invalid_or_missing` (#12) | If `quantized_model_dir` resolved outside `<workspace>`, rewrite it to a relative path under `<workspace>` then re-invoke Step 3 (and re-export if weights were written elsewhere). Otherwise re-invoke `quark-torch-ptq` Step 3. |
| `must_have_config_missing_or_invalid` (#14) | `cp <source_model>/config.json <quantized_model_dir>/`; if vLLM-required keys (`model_type`, `architectures`) are absent, copy them from source's config. Re-run validator Step 3. |
| `must_have_tokenizer_missing` (#15) | `cp <source_model>/tokenizer*` to `<quantized_model_dir>/`. Re-run validator Step 1. |
| `must_validate_config_mismatch` (#17) | Diff config.json source vs quantized after stripping `quantization_config`. Copy the missing/diverged business-field values from source into quantized. Re-run validator Step 3. |
| `should_have_aux_missing` (#19) | `cp` missing auxiliary files (`special_tokens_map.json`, `generation_config.json`, `chat_template.jinja`) from source to quantized. Re-run validator Step 1. |
| `nice_to_have_skipped` (#20) | Record a note (write a `## Notes` line at the bottom of `validation_report.md`); do not retry. |
| `eval_env_unavailable` (#22) | Skip eval; write `<workspace>/eval_skipped.txt` with the reason. |
| `validation_report_absent` (#25) | Re-invoke `quark-torch-result-validator` once. |
| `must_validate_skipped` (#27) | Leave as-is — the Python runner will demote per `HYPERLOOM_QUANT_STRICT_VALIDATION`. |
| `eval_oom` (#29) | Switch eval to serial loading (close source engine before opening quantized). Retry once. If still OOM, write `eval_skipped.txt` with `oom`. |

## 7. Auto-fail catalog (10 hard stops)

For these outcomes, write `outcome_id: <id>` (with a short reason on the next
line) to `<workspace>/blocked.md` and STOP. Do not write a fix hypothesis —
the Python runner will surface the failure to the caller.

| ID | Trigger |
|---|---|
| `quark_root_missing` (#1) | Pre-flight already failed at runner; you should not see this in-session. |
| `exec_model_load_failed` (#4) | Exec step can't load source weights (corrupt safetensors, dtype unsupported, transformers missing). |
| `exec_calibration_data_missing` (#5) | Calibration dataset unreachable or 0 samples. |
| `quark_skill_unavailable` (#7) | A required Quark skill SKILL.md is missing. |
| `model_path_unreachable` (#9) | Intake can't read the source model_path. |
| `validator_self_test_failed` (#13) | `run_validation.py self-test` exits non-zero. |
| `must_validate_md5_mismatch` (#18) | MD5 of an excluded layer differs from source — semantic violation; do NOT retry. |
| `workspace_unwritable` (#23) | Pre-flight already failed; you should not see this in-session. |
| `sdk_runtime_error` (#24) | Pre-flight already failed at runner. |
| `quantized_load_failed` (#28) | vLLM/SGLang can't load the quantized model — equivalent to a MUST-validate failure. |

## 8. Ask + #30 diagnose-fix-retry protocol

These six outcomes (#2 `checkpoint_aborted`, #3 `exec_oom`, #6 `export_crashed`,
#16 `must_have_weights_missing`, #21 `eval_gap_exceeded`, #26
`fuzzy_check_failed`) plus the catch-all #30 `unclassified_failure` MAY
participate in Python-level retry. Before stopping the attempt:

1. **Diagnose**: read stderr / phase context / the relevant validator step.
2. **Write `fix_hypothesis_attempt_<next>.md`** (where `next` = current
   attempt + 1) with the following structure:

   ```markdown
   # Fix hypothesis for attempt N

   ## Root cause
   <one sentence>

   ## Concrete fix
   - <action 1>
   - <action 2>

   ## Risk
   <what could still go wrong>
   ```

   **No "let me try again" placeholders.** If you cannot articulate a
   concrete fix, do NOT write the file — the Python runner uses its absence
   as the gate to skip retry, which is correct behavior.

3. **Write `outcome_id: <id>`** to `<workspace>/blocked.md` so the classifier
   doesn't have to re-discover the cause from disk patterns.

4. **Do not retry yourself** — the Python runner will spawn a new SDK
   session with the fix_hypothesis path in the prompt.

Exceptions:

* `checkpoint_aborted` (#2) and `eval_gap_exceeded` (#21) are decision
  points, not retry candidates. Do NOT write a fix_hypothesis — retrying
  won't synthesize missing information or shrink the gap.

### #30 unclassified_failure decision tree

For any failure that doesn't match #1–#29 (e.g. a Quark upgrade introduced a
new error message):

* If you can patch it in `<workspace>` and try again → write a fix_hypothesis
  and let Python retry.
* If the patch would require editing `quark_root` → write
  `outcome_id: upstream_change_required` to blocked.md and STOP.
* Otherwise → write `outcome_id: unclassified_failure` to blocked.md with
  the traceback summary, and STOP without a fix_hypothesis.

## 9. Workspace file conventions

You may write any of these files; the Python runner reads them:

| File | Owner | Meaning |
|---|---|---|
| `session_context.json` | you (seed) → Quark may augment | Pre-seeded context handed to quark-torch-ptq. |
| `model_analysis.json`  | quark-torch-ptq Step 1 | Source model structural analysis. |
| `quant_plan.json`      | quark-torch-ptq Step 2 | Resolved quant plan. |
| `run_manifest.yaml`    | quark-torch-ptq Step 3 | Includes `outputs.quantized_model_dir`. |
| `validation_report.md` | quark validator  | Per-step ok / FAIL / skipped. |
| `source_eval.md`       | quark-torch-llm-eval (you) | Source model headline metrics. Cached across retries. |
| `quantized_eval.md`    | quark-torch-llm-eval (you) | Quantized model headline metrics. Re-run every attempt. |
| `eval_report.json`     | you | Synthesized wrapper — see §5.3. |
| `eval_gap_threshold.txt` | you (optional) | Per-prompt threshold override. |
| `eval_skipped.txt`     | you | Reason eval bailed (#22 / #28 / #29). |
| `last_phase.txt`       | you | One token: pre/intake/plan/manifest/exec/export/validate/eval. Updated at the START of each phase. |
| `fix_hypothesis_attempt_N.md` | you | Precondition for Python retry — see §8. |
| `blocked.md`           | you | `outcome_id: <id>\n<reason>` for any non-clean stop. |
| `requantize_attempts.txt` | Python | Counter — do NOT touch. |

## 10. Self-check before declaring success

Before the SDK session ends, verify every MUST-have is on disk:

* `run_manifest.yaml` parses; `outputs.quantized_model_dir` resolves to an
  existing directory within `<workspace>` (no `../`, no outside absolute
  path, no escaping symlink). See §1.1.
* That directory contains `config.json`, at least one `*.safetensors` /
  `*.bin`, and at least one tokenizer file (`tokenizer.json` /
  `tokenizer_config.json` / `tokenizer.model` / `vocab.json`).
* `validation_report.md` exists and lists all four steps.
* If eval was requested (it always is unless `eval_skipped.txt` says otherwise):
  `eval_report.json` parses and contains `source_score`, `quantized_score`,
  `relative_gap`.

If any of these is missing and you cannot Auto-recover it per §6, fall
through to §7 or §8 as appropriate. Never declare success when the
self-check fails — the Python classifier will catch it anyway, but a
truthful SKILL.md keeps the attempts log meaningful.