Gptqmodel Lazy Load Smoke Testing

GPT-QModel lazy-load smoke testing

This skill describes how to exercise the lazy checkpoint path on CPU and verify that the first decoder layer is correctly materialized during StageInputsCapture.cache_inputs.

When to use

• A PR touches gptqmodel/looper/module_looper.py, gptqmodel/looper/stage_inputs_capture.py, gptqmodel/models/base.py, or gptqmodel/utils/structure.py.
• You need to verify that shell_module_materialize receives a dotted module_path such as model.layers.0 instead of None or a class name.
• You want to confirm that LazyTurtle.materialize_submodule loads real parameters/buffers (not meta) and that forward/generate runs without AttributeError or NaN.

Devin Secrets Needed

None for CPU-only smoke testing. If GPU testing is required, set CUDA_VISIBLE_DEVICES as appropriate and use device="cuda".

Environment prerequisites

1. Install system development headers; otherwise torch inductor / cpp_extension JIT builds for pack_block_cpu fail with Python.h: No such file or directory:

   sudo apt-get update -qq && sudo apt-get install -y -qq python3-dev python3.10-dev
   

2. Install the CPU torch stack and project dependencies:

   pip install -U pip
   pip install --index-url https://download.pytorch.org/whl/cpu torch torchvision
   pip install -r requirements.txt
   pip install -e . --no-build-isolation --no-deps
   

3. Verify gptqmodel is importable from the repo root:

   python -c "import gptqmodel; print(gptqmodel.__version__)"
   

Minimal fixture

Build a tiny LlamaForCausalLM and save it to a temp directory. The fixture should be small enough to quantize in seconds on CPU:

from transformers import LlamaConfig, LlamaForCausalLM, PreTrainedTokenizerFast
from tokenizers import Tokenizer, models, pre_tokenizers, trainers
import torch, os, tempfile

config = LlamaConfig(
    vocab_size=128,
    hidden_size=64,
    intermediate_size=96,
    num_hidden_layers=1,
    num_attention_heads=4,
    max_position_embeddings=128,
)
model = LlamaForCausalLM(config)
tok = Tokenizer(models.WordLevel(unk_token="<unk>"))
tok.pre_tokenizer = pre_tokenizers.Whitespace()
tok.add_special_tokens(["<s>", "</s>", "<unk>", "<pad>"])
tok_trainer = trainers.WordLevelTrainer(
    special_tokens=["<s>", "</s>", "<unk>", "<pad>"], vocab_size=128
)
tok.train_from_iterator(
    ["the quick brown fox", "lorem ipsum dolor", "hello world"],
    trainer=tok_trainer,
)
hf_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tok, pad_token="<pad>")

tmp = tempfile.mkdtemp()
model.save_pretrained(tmp)
hf_tokenizer.save_pretrained(tmp)

Lazy-load + input-capture test

from gptqmodel import GPTQModel, BACKEND
from gptqmodel.models.base import QUANT_CONFIG_NAME
from gptqmodel.quantization.config import QuantizeConfig

calibration = [
    {"text": "the quick brown fox jumps over the lazy dog"},
    {"text": "lorem ipsum dolor sit amet"},
    {"text": "hello world this is a test"},
]

qcfg = QuantizeConfig(
    bits=4,
    group_size=32,
    desc_act=False,
    device="cpu",
)
model = GPTQModel.load(
    tmp,
    quantize_config=qcfg,
    backend=BACKEND.TORCH,
)
assert model.turtle_model is not None

Capture the materialization path

Wrap model.shell_module_materialize to record the module_path passed during cache_inputs:

calls = []
orig = model.shell_module_materialize

def capture(*args, **kwargs):
    calls.append((args, kwargs))
    return orig(*args, **kwargs)

model.shell_module_materialize = capture
model.quantize(
    calibration,
    batch_size=1,
    backend=BACKEND.GPTQ_TORCH,
    calibration_data_min_length=1,
)

first_call_args, first_call_kwargs = calls[0]
module_path = first_call_kwargs.get("module_path")
assert module_path == "model.layers.0", module_path

Verify first-layer materialization

After model.quantize reaches cache_inputs, inspect the first decoder layer:

first_layer = model.model.model.layers[0]
devs = {p.device.type for p in first_layer.parameters()}
assert "meta" not in devs, f"first layer still has meta parameters: {devs}"
assert not any(torch.isnan(p).any() for p in first_layer.parameters())

Save, reload, and generate

quantized_dir = os.path.join(tmp, "quantized")
model.save(quantized_dir)

qmodel = GPTQModel.load(
    quantized_dir,
    backend=BACKEND.GPTQ_TORCH,
    device="cpu",
)
in_ids = torch.tensor([[2, 5, 11]])
out = qmodel.generate(
    input_ids=in_ids,
    attention_mask=torch.ones_like(in_ids),
    max_new_tokens=4,
    do_sample=False,
)
assert out.shape == (1, in_ids.shape[1] + 4)
assert not torch.isnan(out).any()

Expected log signals

• INFO Loader: using checkpoint-backed lazy turtle source for /tmp/...
• INFO ModuleLooper: capturing layer inputs from N calibration batches
• [test] first materialized module path: 'model.layers.0'
• [test] generate output shape: torch.Size([1, 6])
• [test] PASS

Common issues

• Python.h: No such file or directory during TorchLinear forward: install python3-dev/python3.10-dev and rerun.
• torchao extension load warnings (_C_cutlass_90a.so, _C_mxfp8...so) are non-fatal on CPU.
• Calibration dataset size should be more than 256 warnings are expected for tiny smoke fixtures.
• If module_path is None, the new resolution logic in stage_inputs_capture.py was not reached or layer_names was not propagated from module_looper.py.

Regression test reference

• tests/test_stage_inputs_capture.py covers the module_path resolution paths.
• For end-to-end validation, run smoke_lazy_load.py from the repo root.