Wush Adaptive Quantization Transforms

Summary

WUSH derives closed-form optimal linear blockwise transforms for joint weight-activation quantization in LLMs. The key innovation combines a fixed Hadamard matrix with a data-dependent component derived from second-order statistics of weights and activations, yielding non-orthogonal adaptive transforms that are provably near-optimal for both floating-point and integer quantizers while remaining computationally efficient.

Core Technique

Joint Quantization Optimization: Rather than quantizing weights and activations separately, optimize a transform that works well for both:

x_quantized = Q(T * x)  # T is learned transform, Q is quantizer

Optimal Transform Derivation: The transform minimizes quantization error under the constraint that it's a linear combination:

T = H @ D  # H is fixed Hadamard, D is data-dependent diagonal

The Hadamard component provides rotation invariance, while the diagonal component adapts to data statistics.

Closed-Form Solution: Compute optimal D from:

D_optimal = diag(sqrt(var(weights) @ var(activations)))

using second-order statistics—no iterative optimization needed.

Implementation

Hadamard matrix: Use fixed Walsh-Hadamard matrices (highly structured, GPU-efficient):

import hadamard
H = hadamard.walsh_hadamard_matrix(dim)

Data-dependent diagonal: Compute from training data:

# Collect weight and activation statistics
weight_var = var(all_weights)  # Per-channel variance
act_var = var(all_activations)  # Per-channel variance

# Derive diagonal scaling
D = diag(sqrt(weight_var * act_var))

Transform matrix: Combine components:

T = H @ D @ H.T  # Final transform

Quantization application:

def quantize_with_transform(x, bits=4):
    x_transformed = T @ x
    x_quantized = round_to_bits(x_transformed, bits)
    return inv(T) @ x_quantized

Blockwise application: Apply transforms per block (e.g., 128-dim blocks):

for block_idx in range(0, dim, block_size):
    block = x[block_idx:block_idx+block_size]
    T_block = compute_transform(block)
    x[block_idx:block_idx+block_size] = quantize_with_transform(block, T_block, bits)

When to Use

• Quantizing large language models with memory constraints
• Scenarios requiring both weight and activation quantization
• Applications needing theoretical near-optimality guarantees
• Tasks where GPU-efficient transforms matter for deployment

When NOT to Use

• Scenarios where simple uniform quantization suffices
• Applications where transform overhead is prohibitive
• Models where post-quantization fine-tuning is unavailable
• Real-time inference where quantization computation adds latency

Key References

• Quantization-aware training and post-training quantization
• Linear algebra and matrix factorization
• Hadamard transforms and structured matrices
• Joint optimization for multiple objectives