# Fp8

> Generate a working FP8 quantization example script and save a compressed-tensors checkpoint. Triggers on: "fp8", "FP8_DYNAMIC", "FP8_BLOCK", "MXFP8", "fp8 example", "quantize to fp8".

- Skill: `vllm-project-llm-compressor/fp8` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds add vllm-project-llm-compressor/fp8`
- Raw SKILL.md: https://api.skillmd.com/api/skills/vllm-project-llm-compressor/fp8/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: vllm-project (https://skillmd.com/u/vllm-project-llm-compressor)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/vllm-project-llm-compressor/fp8

---


# Write FP8 Example

Generate a working Python example script that quantizes a model to an FP8 scheme and saves a compressed-tensors checkpoint.

## Step 1 — Gather information

Read the shared documentation at `.claude/skills/shared_quantization.md` for common model information gathering steps, GPTQ, transforms, and calibration dataset configuration.

In addition to the shared information, ask the user (or infer from context) for:

1. **Scheme variant** — choose one:
   - `FP8_BLOCK` — weights fp8 with 128x128 block scaling, activations dynamic. No calibration. Best throughput on Hopper/Blackwell. **Note:** may not be a good choice if the model's weight shapes are not compatible with 128x128 block tiling (e.g. small or irregular hidden dims).
   - `FP8_DYNAMIC` — weights fp8 per-channel, activations fp8 dynamic per-token. No calibration. Broadest hardware support (Ampere+). Use when FP8_BLOCK is not suitable.
   - `MXFP8` — weights and activations in MX fp8 format. No calibration. AMD MI300X target.
2. **Use `model_free_ptq`?** — use `model_free_ptq` if (a) the model does **not** have a definition in the `transformers` library (custom architectures must go through this path), or (b) the model is extremely large (~1TB+) and you want to quantize directly from safetensors files without loading the full model. Otherwise use `oneshot`.

## Templates

Templates are located in `.claude/skills/fp8/templates/`:

- `oneshot.py` — dense-model base template for `oneshot` with `QuantizationModifier`
- `model_free_ptq.py` — template for `model_free_ptq` (no transformers class, or ~1TB+ models)

## Step 2 — Choose the template

### `oneshot` with `QuantizationModifier` (standard path — most common)

This is the most common pathway for FP8 quantization and provides good accuracy without any calibration data. Read `templates/oneshot.py` and use it as the starting point. Apply the model-type adjustments from the shared documentation (`.claude/skills/shared_quantization.md`) before writing the final file.

**Optional: GPTQ or a transform for improved accuracy.** If the user wants to use GPTQ or apply a transform (AWQ, SmoothQuant), calibration data is required. Use the shared template at `.claude/skills/templates/oneshot_with_data.py` instead of `templates/oneshot.py`. Follow the shared documentation to apply GPTQ and/or transform modifications to the recipe.

## Step 3 — Apply model-type adjustments

Apply the model-type adjustments documented in `.claude/skills/shared_quantization.md`.

## Step 4 — `model_free_ptq` (no transformers model definition, or very large models ~1TB+)

Read `templates/model_free_ptq.py` and use it as the starting point. Apply the same `ignore` adjustments from the shared documentation (gate/router layers, vision tower layers) before writing the final file.

## Step 5 — Write the file

Place the file in the appropriate directory under `examples/`:
- `quantization_w8a8_fp8/` for FP8_DYNAMIC or FP8_BLOCK (oneshot path)
- `quantization_w8a8_mxfp8/` for MXFP8
- `model_free_ptq/` when using `model_free_ptq()`

Name the file `{model_name_slug}_example.py` (e.g. `llama3_example.py`, `gemma4_example.py`).

Run `make style` after writing the file.

## Notes
- The standard path (`QuantizationModifier` with no calibration data) is the most common and gives good accuracy for FP8. GPTQ and transforms are optional enhancements — only use them if the user requests improved accuracy.
- `FP8_BLOCK` is preferred for Hopper/Blackwell throughput, but check that the model's weight shapes are compatible with 128x128 block tiling before choosing it.
- `FP8_DYNAMIC` is the fallback when FP8_BLOCK is not suitable — broad hardware support (Ampere+), no calibration required.
- `MXFP8` targets AMD MI300X.
- Neither scheme requires a calibration dataset with plain `QuantizationModifier`; `oneshot(model=model, recipe=recipe)` with no `dataset` argument is correct. Calibration data is only needed when using GPTQ or a transform.
- Use `model_free_ptq` when the model has no transformers class definition, or when the model is ~1TB+ and you want to quantize directly from safetensors without loading the full model.
- `save_compressed=True` is optional — the checkpoint saves in compressed-tensors format either way. Omit unless explicitly requested.