Guoqingbao Xinfer Xinfer

Check Model — Pre-Load Compatibility Audit for xinfer

Phase 0: Gather Model Information

Collect model config and tensor info. Accept any of:

If tensor info is missing, ask the user to provide it. They can get it by clicking any .safetensors file in the HuggingFace model page and copying the tensor tree.

For local models, extract tensor info with:

import json, struct, sys, glob, os
path = sys.argv[1]
for sf in sorted(glob.glob(os.path.join(path, "*.safetensors"))):
    with open(sf, "rb") as f:
        n = struct.unpack("<Q", f.read(8))[0]
        header = json.loads(f.read(n))
    for k, v in sorted(header.items()):
        if k != "__metadata__":
            print(f"{k}\t{v.get('shape')}\t{v.get('dtype')}")

Phase 1: Parse Config and Identify Model Type

Extract from config.json:

Core parameters

Hybrid (Qwen3.5/Qwen3Next) parameters

MoE parameters

Quantization config

Normalize quant_method

Apply the same normalization as QuantConfig::normalize_compressed_tensors():

Phase 2: Validate Tensor Format Against Quantization Config

For each layer type, check that tensor names and dtypes match the expected format.

2a. Determine which layers are quantized vs skipped

Parse the ignore list from quantization_config. Layers in the ignore list should have BF16/FP16 weights (weight tensor only). Layers NOT in the ignore list should have quantized tensors.

The ignore list supports:

• Literal paths: "model.language_model.layers.0.linear_attn.in_proj_qkv"
• Regex patterns: "re:.*linear_attn.*"
• Glob-style wildcards: "model.visual*", "mtp.layers.0*"

2b. Format-specific tensor checks

Unquantized (BF16/FP16)

Expected tensors per linear layer:

• weight — dtype BF16 or F16, shape [out_dim, in_dim]
• bias (optional) — dtype BF16 or F16

Check: No extra scale/packed tensors should be present.

FP8 (quant_method == "fp8")

Expected tensors per linear layer:

• weight — dtype U8 (F8_E4M3), shape [out_dim, in_dim]
• weight_scale or weight_scale_inv — dtype F32, shape [out_dim/by, in_dim/bx] where [by, bx] = weight_block_size (default [128, 128])
• bias (optional)

Check: weight_block_size must have exactly 2 elements. Scale dimensions must match ceil(out_dim/by) x ceil(in_dim/bx).

NVFP4 — ModelOpt format (quant_method == "modelopt" + quant_algo == "NVFP4")

Expected tensors per quantized linear layer:

• weight — dtype U8, shape [out_dim, in_dim/2] (packed FP4, 2 values per byte)
• weight_scale — dtype F8_E4M3 (U8), shape [out_dim, in_dim/16] (group_size=16)
• weight_scale_2 — dtype F32, scalar (global weight scale, direct multiplier)
• input_scale — dtype F32, scalar (activation scale)

Check: weight shape dim1 must be exactly in_dim/2. Scale dim1 must be in_dim/16.

NVFP4 — Compressed-tensors format (quant_method == "compressed-tensors" + nvfp4 format)

Expected tensors per quantized linear layer:

• weight_packed — dtype U8, shape [out_dim, in_dim/2]
• weight_scale — dtype F8_E4M3 (U8), shape [out_dim, in_dim/16]
• weight_global_scale — dtype F32, scalar or [1] (divisor, inverted at load time)
• input_global_scale — dtype F32, scalar or [1] (divisor, inverted at load time)

Check: Same shape rules as ModelOpt, but different tensor names.

MXFP4 (quant_method == "mxfp4" or compressed-tensors with mxfp4)

Expected tensors per quantized linear layer:

• weight_packed or blocks — dtype U8, shape [out_dim, in_dim/2]
• weight_scale or scales — dtype U8 (F8_E8M0), shape [out_dim, in_dim/32] (group_size=32)

Check: Scale dim1 must be in_dim/32.

GGUF

GGUF models are self-contained (no config.json). Weight tensor names use blk.{i} prefix mapped to model.layers.{i}. Quantization is per-tensor via GGML dtypes (Q4_K, Q6_K, Q8_0, etc.).

Check: Not applicable for safetensors checks. GGUF has its own loader path via QLinear / QMatMul.

2c. Loader path tensor name resolution

The xinfer loaders try tensor names in priority order. Verify the model's tensors match at least one:

Flag any mismatch where the model uses a tensor name not in the priority list.

2d. Hybrid model (GDN) quantization detection

For Qwen3.5/Qwen3Next models with quantization config, the GatedDeltaNet layer has its own quantization detection (is_weight_quantized) that checks each linear_attn sublayer independently:

If a linear_attn sublayer is in the ignore list and has only BF16 weight, the detection returns false, and the layer loads as unquantized. Verify this matches the tensor info.

Phase 3: Multi-Rank Divisibility Analysis

For each candidate world_size in [1, 2, 4, 8], check all TP-sharded dimensions.

3a. Full Attention

For quantized (FP8/NVFP4/MXFP4) Q/K/V:

• Column linear shard dim 0: per-rank out_dim / world_size must be cleanly divisible
• For FP8: per-rank start must be aligned to weight_block_size[0] (default 128)
• For NVFP4: per-rank output must be divisible (no block alignment needed for dim 0 shard)

3b. GatedDeltaNet (Linear Attention)

Where:

• key_dim_global = linear_num_key_heads * linear_key_head_dim
• value_dim_global = linear_num_value_heads * linear_value_head_dim

3c. MoE Experts

For NVFP4/MXFP4 MoE:

• gate/up packed dim0: moe_intermediate_size / world_size per rank
• down packed dim1: (moe_intermediate_size / pack_factor) / world_size per rank

3d. Shared Expert MLP

Same rules as standard MLP with shared_expert_intermediate_size:

• Column linear (gate/up): shared_expert_intermediate_size % world_size == 0
• Row linear (down): shared_expert_intermediate_size % world_size == 0

3e. NVFP4/MXFP4 Scale Alignment

For NVFP4 (group_size=16): after sharding, verify per_rank_in_dim % 16 == 0 for dim-1 shards. For MXFP4 (group_size=32): verify per_rank_in_dim % 32 == 0 for dim-1 shards. For FP8: verify per-rank boundaries align to weight_block_size.

3f. Embedding / LM Head

• embed_tokens: replicated (not sharded), no divisibility constraint.
• lm_head: replicated, no constraint. But if tie_word_embeddings is true, verify lm_head doesn't exist as a separate tensor (should reuse embed_tokens.weight).

Phase 4: Report Findings

Present results in a structured format:

Model Summary

Architecture: Qwen3_5MoeForConditionalGeneration
Model Type: qwen3_5_moe (Hybrid MoE with linear attention)
Quantization: nvfp4 (compressed-tensors format)
Layers: 48 (36 linear_attention + 12 full_attention)
Hidden size: 3072
Full attention: 32 Q heads, 2 KV heads, head_dim=256
Linear attention: 16 K heads, 64 V heads, head_dim=128
MoE: 256 experts, top-8, intermediate=1024
Shared expert: intermediate=1024

Tensor Format Validation

[OK] Linear attention layers (BF16, in ignore list)
[OK] Full attention layers (NVFP4 compressed-tensors: weight_packed + weight_scale + weight_global_scale)
[OK] MoE experts (NVFP4 compressed-tensors: per-expert weight_packed)
[OK] Shared expert MLP (NVFP4 compressed-tensors: weight_packed)
[WARN] lm_head: in ignore list, stored as BF16

Multi-Rank Compatibility

| Component | 1 GPU | 2 GPUs | 4 GPUs | 8 GPUs |
|-----------|-------|--------|--------|--------|
| Full attn Q heads (32) | OK | 16 | 8 | 4 |
| Full attn KV heads (2) | OK | 1 | repl(2) | repl(4) |
| GDN K heads (16) | OK | 8 | 4 | 2 |
| GDN V heads (64) | OK | 32 | 16 | 8 |
| MoE inter (1024) | OK | 512 | 256 | 128 |
| Overall | OK | OK | OK | OK |

Issues Found

Flag any problems:

• [ERROR] — Will fail to load (missing tensors, wrong names, indivisible dims)
• [WARN] — May cause issues (unusual format, edge case)
• [INFO] — Informational (features detected, fallback paths)

Phase 5: Common Issues Reference

Known tensor name mismatches

GatedDeltaNet TP-safe loading

The in_proj_qkv tensor requires special merged-chunk loading for multi-GPU:

• MergedParallelColumnLinear::load_merged_chunks splits Q, K, V independently
• For quantized models (FP8/NVFP4/MXFP4), each chunk must be sharded within the quantized domain
• For BF16 (ignore-listed layers), falls through to the unquantized path

Replicated KV heads

When num_kv_heads < world_size:

• kv_head_shard uses replicated mode: ranks_per_kv_head = world_size / num_kv_heads
• Each KV head is shared by ranks_per_kv_head consecutive ranks
• Requires world_size % num_kv_heads == 0

Key Source Files

Source: guoqingbao/xinfer — distributed by TomeVault.