Gendeg Eval

gendeg-eval

GenDeg: Diffusion-based Degradation Synthesis for Generalizable All-In-One Image Restoration — Sudarshan Rajagopalan et al. (arXiv:2411.17687, 2024)

What this evaluates

Evaluates the out-of-distribution (OoD) generalization and within-distribution performance of All-In-One Image Restoration (AIOR) models across six degradation types (haze, rain, snow, motion blur, raindrop, low-light) when trained with synthetic degradation data.

Datasets

• O-HAZE — total ?; splits: test (-1)
• LHP — total ?; splits: test (-1)
• RainDS — total ?; splits: test (-1)
• RSVD — total ?; splits: test (-1)
• GoPro — total ?; splits: test (-1)

Metrics

• LPIPS (primary) — range: [0, 1]
  • Learned Perceptual Image Patch Similarity. Measures perceptual difference between restored and ground truth images. Lower is better.
• FID — range: [0, inf)
  • Fréchet Inception Distance. Measures the distance between feature distributions of restored and ground truth images. Lower is better.

Input / output format

Input: Degraded images corresponding to one of six degradation types (haze, rain, snow, motion blur, raindrop, low-light).

Output: Restored images.

Scoring recipe

def evaluate(restored_imgs, gt_imgs):
    lpips_vals = [lpips_metric(r, g) for r, g in zip(restored_imgs, gt_imgs)]
    fid_val = frechet_inception_distance(restored_imgs, gt_imgs)
    return mean(lpips_vals), fid_val

Common pitfalls

• PSNR and SSIM scores can be misleadingly low because restored images often exhibit richer colors than the ground truth, despite better perceptual quality.
• OoD generalization is the primary evaluation goal, but within-distribution performance must also be checked to ensure no catastrophic forgetting or degradation occurs when adding synthetic data.
• Motion blur results may show minimal improvement from synthetic data augmentation due to the high diversity already present in real motion blur datasets.

Evidence (verbatim from paper)

Quantitative comparisons. Due to space constraints, we present quantitative comparisons using only the LPIPS and FID metrics (following[[29]]). Table[2] presents these scores for OoD test sets across all six degradations. ... Notably, the enhanced images often contain richer colors than the ground truth (see first row), which can lower PSNR and SSIM scores. Thus, LPIPS and FID scores are more reliable for testing the OoD performance.

Citation

@misc{rajagopalan2024gendeg,
  title={GenDeg: Diffusion-based Degradation Synthesis for Generalizable All-In-One Image Restoration},
  author={Sudarshan Rajagopalan et al.},
  year={2024},
  note={arXiv:2411.17687}
}

• arXiv: 2411.17687