# Machin Diffusion Validation

> How to validate output correctness after optimizations. Reference comparison, pixel diff methodology, and image quality metrics.

- Skill: `javimosch/machin-diffusion-validation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add javimosch/machin-diffusion-validation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/javimosch/machin-diffusion-validation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: javimosch (https://skillmd.com/u/javimosch)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/javimosch/machin-diffusion-validation

---


# Validation Methodology

## Why validate after every optimization

Fused GPU kernels reorder floating-point operations, giving slightly different results. The goal is to confirm the optimization didn't break correctness — not to achieve bit-exact reproduction.

## Reference pipeline

The reference is at `/tmp/ref_pipeline.py` (diffusers/PyTorch with the **correct trailing scheduler**). It validates stage-by-stage:

| Stage | Metric | Typical value |
|-------|--------|--------------|
| CLIP embeddings | max diff | ~0.000018 |
| UNet noise prediction | max diff | ~0.000307 |
| UNet noise prediction | correlation | 1.000000 |
| Final latent | max diff | ~0.000070 |
| Final latent | correlation | 1.000000 |
| Image | mean pixel diff | 0.01–0.05 |
| Image | max pixel diff | 1–2 |

After fused kernels (group_norm_silu, attention_f32, conv3x3_t4x4, matmul_tiled):
- **pixel diff mean=0.02, max=1** — within tolerance, from float reordering in online softmax and tiled accumulation.

## Standard validation prompt

```
"a girl photo, close take"
seed: 12345
guidance_scale: 0.0
512x512
trailing scheduler
```

Expected output: photo-like image, ~75% skin-tone pixels, edge strength ~5.0, R mean ~151.

## How to compare images

```python
import numpy as np
from PIL import Image

prev = np.array(Image.open('previous.png'), dtype=np.int16)
curr = np.array(Image.open('current.png'), dtype=np.int16)
diff = np.abs(curr - prev)
print(f'pixel diff mean={diff.mean():.2f} max={diff.max()}')
# Acceptable: mean < 0.05, max <= 1
```

## Image quality metrics

```python
arr = np.array(img)
print(f'R: mean={arr[:,:,0].mean():.0f} std={arr[:,:,0].std():.0f}')
gray = arr.mean(axis=2).astype(np.float32)
gx = np.abs(np.diff(gray, axis=1))
print(f'edge strength: {gx.mean():.1f}')
skin = ((arr[:,:,0] > arr[:,:,1]) & (arr[:,:,1] > arr[:,:,2]) &
       (arr[:,:,0] > 80) & (arr[:,:,0] < 220)).sum()
print(f'skin-tone: {100*skin/(w*h):.1f}%')
```

## Fetching output from Windows

```bash
# PPM → base64 → decode locally
rcx ordi-jla "certutil -encode output.ppm out.b64 && type out.b64" 60 > /tmp/raw
# Then: decode base64, parse PPM header (P6\n<w> <h>\n255\n), load with PIL
```

## What "correct" looks like

- **Abstract noise** → scheduler is wrong (check `timestep_spacing: "trailing"`)
- **High-contrast garbage** → CFG was added (check `guidance_scale: 0.0`)
- **Photo-like, warm, smooth** → correct
- **pixel diff > 0.05 or max > 2** → optimization broke something, investigate

## Don't claim byte-identical

After fused kernels, output is numerically equivalent but not bit-identical. Use measured tolerances (mean < 0.05, max ≤ 1) in documentation, not "byte-identical" or "exact".

## SDXL-Lightning validation (2026-09-08)

### Critical finding: CLIP attention must be CAUSAL

CLIP text encoders (both TE1 and TE2) use **causal** attention — position i only attends
to positions j ≤ i. The SDXL port originally used bidirectional `attention_f32`, which
produced TE2 hidden states with only **0.18 correlation** to the diffusers reference.
This was the root cause of the abstract noise output.

**Fix:** Added `attention_causal_f32` builtin to the machin compiler (OpenCL kernel +
CPU fallback). After the fix, TE2 hidden correlation jumped to **0.9993**.

### SDXL reference setup (RunPod)

The reference pipeline (`scripts/sdxl_reference.py`) loads components directly:
- `CLIPTextModel` (TE1) + `CLIPTextModelWithProjection` (TE2) with `attn_implementation="eager"`
- `EulerDiscreteScheduler` with `timestep_spacing="trailing"`, 4 timesteps
- `UNet2DConditionModel` loaded from `unet_4step_fp32.safetensors`
- `AutoencoderKL` with `scaling_factor=0.13025`

**Package compatibility:** torch 2.5.1+cu121, transformers 4.48.3, diffusers 0.40.0,
huggingface_hub 1.30.0. Earlier versions had import errors (CLIPImageProcessor removed
in transformers 5.x, `get_cached_repo_tree` missing in old hub, SDPA causal mask shape bug).

### Validated stages

| Stage | Metric | Value |
|-------|--------|-------|
| TE2 hidden [77×1280] | correlation | 0.9993 |
| TE2 hidden | mean_diff | 0.019 |
| Scheduler sigmas | exact | [14.615, 4.082, 1.613, 0.693, 0.0] |
| Scheduler timesteps | exact | [999, 749, 499, 249] |
| VAE scaling factor | exact | 0.13025 |

### Pending validation (pod terminated, low balance)

- UNet step-0 noise prediction
- VAE decode output
- Pooled output after text_projection
- Visual image comparison

### Debug dump format

`main_sdxl.src` accepts an optional 4th argument `debug_dir`. When provided, dumps:
- `te2_hidden.bin` — [77, 1280] float32
- `te2_pooled_raw.bin` — [1280] float32 (before text_projection)
- `vae_output.bin` — [3, 1024, 1024] float32

The reference saves `.npy` files for all stages. Compare with:
```python
ref = np.load('/workspace/ref/te2_hidden.npy')[0]  # [77, 1280]
ours = np.fromfile('/workspace/our/te2_hidden.bin', dtype=np.float32).reshape(77, 1280)
corr = np.corrcoef(ref.flatten(), ours.flatten())[0,1]
```