Diffusion Rep Eval

diffusion-rep-eval

Do Diffusion Models Learn Semantically Meaningful and Efficient Representations? — Liang et al. (2024) (arXiv:2402.03305, 2024)

What this evaluates

Evaluates whether conditional diffusion models learn semantically meaningful and factorized representations by measuring generation accuracy against ground truth latent coordinates and the predictive power of internal model embeddings over those coordinates.

Datasets

• Synthetic 2D Gaussian Bump Dataset — total ?; splits: train (-1), test (-1)

Metrics

• predicted label accuracy (primary) — range: [0, 1]
  • Accuracy of the model's generated images in matching the ground truth latent coordinates (mu_x, mu_y), normalized to [0, 1].
• averaged R-squared — range: [0, 1]
  • Coefficient of determination (R^2) measuring how well a linear model predicts the ground truth labels (mu_x, mu_y) from the model's 3D-reduced internal representation, averaged across dimensions.

Input / output format

Input: A 32x32 grayscale image containing a single 2D Gaussian bump, conditioned on its ground truth center coordinates (mu_x, mu_y).

Output: Generated image(s) matching the condition, and internal layer-4 embeddings reduced to 3D for representation analysis.

Scoring recipe

def compute_metrics(predictions, gold):
    # predicted label accuracy
    matches = sum(1 for p, g in zip(predictions, gold) if np.allclose(p['center'], g['center'], atol=tol))
    accuracy = matches / len(gold)
    
    # averaged R-squared
    embeddings = get_layer4_embeddings(predictions)
    emb_3d = reduce_dim(embeddings, n_components=3)
    r2_x = r2_score(gold['mu_x'], linear_model.predict(emb_3d)[:, 0])
    r2_y = r2_score(gold['mu_y'], linear_model.predict(emb_3d)[:, 1])
    r2_avg = np.mean([r2_x, r2_y])
    return accuracy, r2_avg

Common pitfalls

• The dataset is synthetic and parameterized by increments dx, dy and spread sigma, so results vary significantly with data density and overlap.
• Exact tolerance thresholds for accuracy and regression details are deferred to Appendix B.3, not fully specified in the main text.
• Internal representations are reduced to 3D for visualization/analysis, which may lose information compared to the full UNet bottleneck.

Evidence (verbatim from paper)

To briefly summarize, we assess the performance of the model based on the accuracies of the images generated and the quality of fit of the 3D embedding of the internal representation corresponding to the sampled images in predicting the ground truth image labels. We refer to the two quantitative performance indicators as the predicted label accuracy and the averaged R-squared. Intuitively, these two metrics range from 0 to 1, with the closer they are to 1 the higher quality of the generated images/learned representation, i.e., the better the performance of the model.

Citation

@misc{liang2024diffusion,
  title={Do Diffusion Models Learn Semantically Meaningful and Efficient Representations?},
  author={Liang et al. (2024)},
  year={2024},
  note={arXiv:2402.03305}
}

• arXiv: 2402.03305