# Hf Gguf Quant Jobs > Use when creating, monitoring, validating, or documenting low-memory Hugging Face Jobs or local runs that quantize split BF16/FP16 GGUF model repos into custom quant GGUF repos with skippy-quantize. - Skill: `mesh-llm/hf-gguf-quant-jobs` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add mesh-llm/hf-gguf-quant-jobs` - Raw SKILL.md: https://api.skillmd.com/api/skills/mesh-llm/hf-gguf-quant-jobs/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: mesh-llm (https://skillmd.com/u/mesh-llm) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/mesh-llm/hf-gguf-quant-jobs --- # HF GGUF Quant Jobs Use this skill to turn an existing split BF16/FP16 GGUF model repo into a quantized GGUF model repo without requiring the host to hold the full model in memory or on local disk at once. The operational tool is `skippy-quantize`; do not use `llama-quantize`, `llama-quantise`, or wrapper scripts that shell out to those binaries. The supported pattern is: mount or point at the source BF16/FP16 GGUF repo, quantize resumable split windows with `skippy-quantize`, publish completed output shards to the target model repo, delete staged files immediately, and resume from the first missing target shard after cancellation or failure. ## Preconditions - Use a split BF16/FP16 GGUF repo as the source when possible. Do not re-read SafeTensors for requants if a BF16 GGUF artifact already exists. - Verify the source repo is complete before spending on quantization. Count all expected split shards and refuse to run if any are missing. - Use a tensor-type file for any custom recipe. Treat MTP tensors, output tensors, precision-sensitive tensors, and latency-sensitive layer ranges as explicit recipe inputs. - Run jobs under the intended HF org and pass `HF_TOKEN` as a secret, not a printed environment variable. - Prefer mounted Hub repos over full `hf download` when the job only needs to stream or stage one shard/window at a time. - Build the standalone binary with `just skippy-quantize-standalone-release-build` for local runs or in the job image/script for HF Jobs. ## Workflow 1. Identify the source BF16/FP16 GGUF repo, target quant repo, output prefix, output basename, source prefix, quant type, tensor-type file, memory budget, and split window size. 2. Preflight both Hub and mounted source paths with `skippy-quantize status`, `next-window`, `validate-splits`, or a `quantize --preflight-only` run. Stop if the source artifact is incomplete. 3. Write or upload a `quant-plan.json` with source repo/revision, target repo, quant type, shard count, output prefix, tensor policy, and resume settings. 4. Launch the job with `--window-size 1` for the first full model run unless a smaller fixture proves a larger window is safe on the chosen hardware. 5. For each split window, stage only the required input shard, run `skippy-quantize run-quant-window` or `run-quant`, publish finished shards, then delete local staged input and output files. 6. Monitor for progress markers. A healthy job repeatedly emits staged source copies, `quant_window`, publish completion, cleanup, and increasing split progress. 7. Validate the target repo after completion by counting GGUF shards, checking the first and last shard names, and confirming `quant-plan.json` plus the tensor-type file are present. 8. Record the artifact in the experiment card and create an iteration card for the run, including job id, command, environment, repo SHA, shard count, and follow-up decisions. ## Launch Template Create a quantization manifest: ```bash target/release/skippy-quantize init-quant \ --source /mnt/source-gguf \ --source-prefix \ --target /mnt/target-quant \ --target-prefix \ --output-basename \ --quant \ --tensor-type-file /mnt/recipe/tensor-types.txt \ --window-size 1 \ --manifest /tmp/skippy-quantize.json ``` Dry-run the next quantization window before spending I/O: ```bash target/release/skippy-quantize quant-job \ --source /mnt/source-gguf \ --source-prefix \ --target /mnt/target-quant \ --target-prefix \ --output-basename \ --quant \ --tensor-type-file /mnt/recipe/tensor-types.txt \ --window-size 1 \ --manifest /tmp/skippy-quantize.json \ --backend llama-api \ --max-memory 32G \ --dry-run ``` Run until complete: ```bash target/release/skippy-quantize run-quant \ --manifest /tmp/skippy-quantize.json \ --backend llama-api \ --max-memory 32G \ --work-dir /tmp/skippy-quantize-work \ --spool-dir /tmp/skippy-quantize-output \ --record-dir /tmp/skippy-quantize-records \ --json-event-file /tmp/skippy-quantize-status.json \ --json-event-interval-seconds 120 \ --json-event-window 8 ``` For HF Jobs, mount the BF16/FP16 source repo and target quant repo, then run the same manifest and `run-quant` commands inside the job: ```bash hf jobs uv run \ --namespace meshllm \ --flavor cpu-upgrade \ --timeout 3d \ --secrets HF_TOKEN \ --volume hf://models/:/mnt/source-gguf \ --volume hf://models/:/mnt/target-quant \ --env SKIPPY_QUANTIZE_OUTPUT=json \ --env PYTHONUNBUFFERED=1 \ --detach \ /path/to/skippy_quant_job.py \ -- \ --source /mnt/source-gguf \ --source-prefix \ --target /mnt/target-quant \ --target-prefix \ --output-basename \ --quant \ --tensor-type-file /mnt/recipe/tensor-types.txt \ --max-memory 32G ``` The job script should only build or install `skippy-quantize`, prepare the manifest if missing, run `run-quant`, verify the job, and upload sidecars. ## Monitoring Check status and logs: ```bash hf jobs inspect --namespace meshllm hf jobs logs --namespace meshllm --tail 120 ``` For agents, prefer polling `/tmp/skippy-quantize-status.json` over ingesting full logs. It is a periodically refreshed compact snapshot with the current phase, current split window, and a bounded recent-event window. Useful healthy markers: - `Preflight QuantizeGguf with backend llama-api` - `Source artifact is complete` - `quant_window` - `Published /mnt/target-quant/...` - `Cleaned staged source` - `split artifact ... 100.00%` Concerning markers: - repeated watchdog lines with no shard, tensor, upload, or cache-drop progress; - cgroup memory pinned near the hardware limit; - the same split window restarting repeatedly without new uploaded target files; - fallback quant warnings for tensors that the recipe expected to preserve. If a job stalls, cancel it before changing code or hardware. The next run should skip already published shards and resume at the first missing output shard. ## Validation After completion, verify the target repo with an authenticated Hub API or CLI check. Record at least: - target repo and commit SHA; - privacy setting; - total file count; - GGUF shard count; - first and last shard names; - manifest/plan presence; - tensor-type file presence. For local smoke tests, use a small split GGUF source first and verify: - `skippy-quantize verify-job --manifest --llama-load` succeeds; - `skippy-quantize validate-splits --root --prefix ` succeeds; - max RSS stays bounded compared with full-model size; - `skippy-quantize status --manifest --json` reports completion. ## Documentation Contract For Jianyang-style experiments, update both records: - the main experiment card with the promoted artifact; - a phase iteration card with the job id, exact command, environment, verification output, decision, and follow-ups. Keep post-experiment upstream notes separate from the run decision. The job can be successful while the converter or quantizer patches still need extraction into clean upstream PRs.