# Diffusion Rep Eval

> Evaluates whether conditional diffusion models learn semantically meaningful and factorized representations by measuring generation accuracy against ground truth latent coordinates and the predictive power of internal model embeddings over those coordinates. Use when the user wants to benchmark on Synthetic 2D Gaussian Bump Dataset, or asks about evaluating this task. Reports predicted label accuracy.

- Skill: `qhjqhj00/diffusion-rep-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/diffusion-rep-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/diffusion-rep-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/diffusion-rep-eval

---


# diffusion-rep-eval

> Do Diffusion Models Learn Semantically Meaningful and Efficient Representations? — Liang et al. (2024) (arXiv:2402.03305, 2024)

## What this evaluates

Evaluates whether conditional diffusion models learn semantically meaningful and factorized representations by measuring generation accuracy against ground truth latent coordinates and the predictive power of internal model embeddings over those coordinates.

## Datasets

- **Synthetic 2D Gaussian Bump Dataset** — total ?; splits: train (-1), test (-1)

## Metrics

- `predicted label accuracy` **(primary)** — range: [0, 1]
  - Accuracy of the model's generated images in matching the ground truth latent coordinates (mu_x, mu_y), normalized to [0, 1].
- `averaged R-squared` — range: [0, 1]
  - Coefficient of determination (R^2) measuring how well a linear model predicts the ground truth labels (mu_x, mu_y) from the model's 3D-reduced internal representation, averaged across dimensions.

## Input / output format

**Input**: A 32x32 grayscale image containing a single 2D Gaussian bump, conditioned on its ground truth center coordinates (mu_x, mu_y).

**Output**: Generated image(s) matching the condition, and internal layer-4 embeddings reduced to 3D for representation analysis.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # predicted label accuracy
    matches = sum(1 for p, g in zip(predictions, gold) if np.allclose(p['center'], g['center'], atol=tol))
    accuracy = matches / len(gold)
    
    # averaged R-squared
    embeddings = get_layer4_embeddings(predictions)
    emb_3d = reduce_dim(embeddings, n_components=3)
    r2_x = r2_score(gold['mu_x'], linear_model.predict(emb_3d)[:, 0])
    r2_y = r2_score(gold['mu_y'], linear_model.predict(emb_3d)[:, 1])
    r2_avg = np.mean([r2_x, r2_y])
    return accuracy, r2_avg
```

## Common pitfalls

- The dataset is synthetic and parameterized by increments dx, dy and spread sigma, so results vary significantly with data density and overlap.
- Exact tolerance thresholds for accuracy and regression details are deferred to Appendix B.3, not fully specified in the main text.
- Internal representations are reduced to 3D for visualization/analysis, which may lose information compared to the full UNet bottleneck.

## Evidence (verbatim from paper)

> To briefly summarize, we assess the performance of the model based on the accuracies of the images generated and the quality of fit of the 3D embedding of the internal representation corresponding to the sampled images in predicting the ground truth image labels. We refer to the two quantitative performance indicators as the predicted label accuracy and the averaged R-squared. Intuitively, these two metrics range from 0 to 1, with the closer they are to 1 the higher quality of the generated images/learned representation, i.e., the better the performance of the model.

## Citation

```bibtex
@misc{liang2024diffusion,
  title={Do Diffusion Models Learn Semantically Meaningful and Efficient Representations?},
  author={Liang et al. (2024)},
  year={2024},
  note={arXiv:2402.03305}
}
```

- arXiv: 2402.03305