Machin Diffusion Validation

Validation Methodology

Why validate after every optimization

Fused GPU kernels reorder floating-point operations, giving slightly different results. The goal is to confirm the optimization didn't break correctness — not to achieve bit-exact reproduction.

Reference pipeline

The reference is at /tmp/ref_pipeline.py (diffusers/PyTorch with the correct trailing scheduler). It validates stage-by-stage:

After fused kernels (group_norm_silu, attention_f32, conv3x3_t4x4, matmul_tiled):

• pixel diff mean=0.02, max=1 — within tolerance, from float reordering in online softmax and tiled accumulation.

Standard validation prompt

"a girl photo, close take"
seed: 12345
guidance_scale: 0.0
512x512
trailing scheduler

Expected output: photo-like image, ~75% skin-tone pixels, edge strength ~5.0, R mean ~151.

How to compare images

import numpy as np
from PIL import Image

prev = np.array(Image.open('previous.png'), dtype=np.int16)
curr = np.array(Image.open('current.png'), dtype=np.int16)
diff = np.abs(curr - prev)
print(f'pixel diff mean={diff.mean():.2f} max={diff.max()}')
# Acceptable: mean < 0.05, max <= 1

Image quality metrics

arr = np.array(img)
print(f'R: mean={arr[:,:,0].mean():.0f} std={arr[:,:,0].std():.0f}')
gray = arr.mean(axis=2).astype(np.float32)
gx = np.abs(np.diff(gray, axis=1))
print(f'edge strength: {gx.mean():.1f}')
skin = ((arr[:,:,0] > arr[:,:,1]) & (arr[:,:,1] > arr[:,:,2]) &
       (arr[:,:,0] > 80) & (arr[:,:,0] < 220)).sum()
print(f'skin-tone: {100*skin/(w*h):.1f}%')

Fetching output from Windows

# PPM → base64 → decode locally
rcx ordi-jla "certutil -encode output.ppm out.b64 && type out.b64" 60 > /tmp/raw
# Then: decode base64, parse PPM header (P6\n<w> <h>\n255\n), load with PIL

What "correct" looks like

• Abstract noise → scheduler is wrong (check timestep_spacing: "trailing")
• High-contrast garbage → CFG was added (check guidance_scale: 0.0)
• Photo-like, warm, smooth → correct
• pixel diff > 0.05 or max > 2 → optimization broke something, investigate

Don't claim byte-identical

After fused kernels, output is numerically equivalent but not bit-identical. Use measured tolerances (mean < 0.05, max ≤ 1) in documentation, not "byte-identical" or "exact".

SDXL-Lightning validation (2026-09-08)

Critical finding: CLIP attention must be CAUSAL

CLIP text encoders (both TE1 and TE2) use causal attention — position i only attends to positions j ≤ i. The SDXL port originally used bidirectional attention_f32, which produced TE2 hidden states with only 0.18 correlation to the diffusers reference. This was the root cause of the abstract noise output.

Fix: Added attention_causal_f32 builtin to the machin compiler (OpenCL kernel + CPU fallback). After the fix, TE2 hidden correlation jumped to 0.9993.

SDXL reference setup (RunPod)

The reference pipeline (scripts/sdxl_reference.py) loads components directly:

• CLIPTextModel (TE1) + CLIPTextModelWithProjection (TE2) with attn_implementation="eager"
• EulerDiscreteScheduler with timestep_spacing="trailing", 4 timesteps
• UNet2DConditionModel loaded from unet_4step_fp32.safetensors
• AutoencoderKL with scaling_factor=0.13025

Package compatibility: torch 2.5.1+cu121, transformers 4.48.3, diffusers 0.40.0, huggingface_hub 1.30.0. Earlier versions had import errors (CLIPImageProcessor removed in transformers 5.x, get_cached_repo_tree missing in old hub, SDPA causal mask shape bug).

Validated stages

Pending validation (pod terminated, low balance)

• UNet step-0 noise prediction
• VAE decode output
• Pooled output after text_projection
• Visual image comparison

Debug dump format

main_sdxl.src accepts an optional 4th argument debug_dir. When provided, dumps:

• te2_hidden.bin — [77, 1280] float32
• te2_pooled_raw.bin — [1280] float32 (before text_projection)
• vae_output.bin — [3, 1024, 1024] float32

The reference saves .npy files for all stages. Compare with:

ref = np.load('/workspace/ref/te2_hidden.npy')[0]  # [77, 1280]
ours = np.fromfile('/workspace/our/te2_hidden.bin', dtype=np.float32).reshape(77, 1280)
corr = np.corrcoef(ref.flatten(), ours.flatten())[0,1]