Dc Ae Recon Gen Eval

dc-ae-recon-gen-eval

Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models — Chen et al. (2024) (arXiv:2410.10733, 2024)

What this evaluates

Evaluates the reconstruction fidelity of high-spatial-compression autoencoders and the generation quality and efficiency of latent diffusion models that utilize them. It benchmarks performance across multiple datasets and resolutions to assess trade-offs between compression ratio, image quality, and computational throughput.

Datasets

• ImageNet — total ?; splits: train (-1)
• FFHQ — total ?; splits: test (-1)
• MapillaryVistas — total ?; splits: test (-1)
• MJHQ — total ?; splits: test (-1)

Metrics

• rFID (primary) — range: other
  • Fréchet Inception Distance computed between the feature distributions of original and reconstructed images. Lower values indicate better reconstruction fidelity.
• FID (primary) — range: other
  • Fréchet Inception Distance computed between generated images and real images. Lower values indicate better generation quality.
• PSNR — range: other
  • Peak Signal-to-Noise Ratio in decibels, measuring pixel-wise reconstruction error. Higher values indicate better quality.
• SSIM — range: [0, 1]
  • Structural Similarity Index, measuring perceptual similarity in luminance, contrast, and structure. Range [0, 1].
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity, measuring perceptual distance using deep features. Range [0, 1].
• CLIP Score — range: other
  • Cosine similarity between CLIP image and text embeddings. Higher values indicate better alignment.

Input / output format

Input: Original images for reconstruction tasks; class labels or text prompts for generation tasks.

Output: Reconstructed images or generated images.

Scoring recipe

def evaluate(original_imgs, reconstructed_imgs, generated_imgs=None, prompts=None):
    rfid = compute_fid(original_imgs, reconstructed_imgs)
    psnr = compute_psnr(original_imgs, reconstructed_imgs)
    ssim = compute_ssim(original_imgs, reconstructed_imgs)
    lpips = compute_lpips(original_imgs, reconstructed_imgs)
    metrics = {'rFID': rfid, 'PSNR': psnr, 'SSIM': ssim, 'LPIPS': lpips}
    if generated_imgs is not None:
        fid = compute_fid(generated_imgs, original_imgs)
        clip = compute_clip_similarity(generated_imgs, prompts)
        metrics['FID'] = fid
        metrics['CLIP Score'] = clip
    return metrics

Common pitfalls

• rFID measures autoencoder reconstruction quality, not generative model performance; confusing it with standard FID leads to incorrect conclusions about generation capability.
• Comparisons across autoencoders must account for patch size (p1 vs p2/p4) and latent channel count, as these drastically change token count, memory, and throughput.
• NFE (Number of Functional Evaluations) is fixed for most models but adaptive for SiT/USiT, making throughput/latency comparisons sensitive to the evaluation scheduler used.

Evidence (verbatim from paper)

Table 2 summarizes the results of DC-AE and SD-VAE (Rombach et al., [2022]) under various settings (f represents the spatial compression ratio and c denotes the number of latent channels). DC-AE provides significant reconstruction accuracy improvements than SD-VAE for all cases. For example, on ImageNet 512×512, DC-AE improves the rFID from 16.84 to 0.22 for the f64c128 autoencoder and 100.74 to 0.23 for the f128c512 autoencoder.

Citation

@misc{chen2024deepcompressionautoencoder,
  title={Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2410.10733}
}

• arXiv: 2410.10733