Georemover Eval

georemover-eval

GeoRemover: Removing Objects and Their Causal Visual Artifacts — Zhu et al. (2025) (arXiv:2509.18538, 2025)

What this evaluates

Evaluates the quality of object removal and causal visual artifact removal (shadows, reflections) in images, measuring visual fidelity, structural consistency, and artifact suppression.

Datasets

• RORD-Val — total ?; splits: val (-1)
• RemovalBench — total ?; splits: test (-1)
• CausRem — total 200; splits: test (200)

Metrics

• FID (primary) — range: other
  • Fréchet Inception Distance; measures the Wasserstein-2 distance between Gaussian distributions of Inception features for real and generated images.
• CMMD — range: other
  • CLIP Maximum Mean Discrepancy; measures distribution discrepancy between real and generated images in CLIP embedding space.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity; computes perceptual distance using deep network features.
• PSNR — range: other
  • Peak Signal-to-Noise Ratio; measures pixel-level fidelity in decibels.
• AS — range: other
  • Aesthetic Score; predicts human aesthetic preference using a pre-trained model.
• MAE — range: other
  • Mean Absolute Error; computes average depth prediction error within the masked object region.
• IoU% — range: [0, 1]
  • Intersection over Union; measures overlap between ground-truth artifact masks and predicted residue regions (identified via a fixed pixel-difference threshold of 20).

Input / output format

Input: Masked RGB image and corresponding depth map, along with a fixed text prompt ('a beautiful scene').

Output: Synthesized RGB image with the target object and associated causal visual artifacts removed.

Scoring recipe

# For FID/CMMD/LPIPS/PSNR/AS:
scores = compute_metrics(predictions, ground_truth_images)
# For MAE (depth):
mae = np.mean(np.abs(pred_depth[mask] - gt_depth[mask]))
# For IoU% (artifacts):
residue = np.abs(pred_img - gt_img) > 20
iou = np.sum(residue & artifact_mask) / np.sum(artifact_mask)
# For Insertion rate:
insert_rate = count_wrong_insertions / total_samples

Common pitfalls

• FID and CMMD are aggregate distribution metrics that can hide severe per-image failures or hallucinations.
• IoU for artifact removal uses a fixed global pixel-difference threshold (20), which may misclassify residue in varying lighting conditions.
• Aesthetic Score (AS) is subjective and can be artificially inflated by over-smoothing or texture hallucination.

Evidence (verbatim from paper)

We use both RORD-Val and RemovalBench as our primary benchmarks. Moreover, we follow prior works and adopt a set of metrics to evaluate image generation quality. We use Frechet Inception Distance (FID), CLIP Maximum Mean Discrepancy (CMMD), Aesthetic Score (AS), Learned Perceptual Image Patch Similarity (LPIPS) and Peak Signal-to-Noise Ratio (PSNR).

Citation

@misc{zhu2025georemover,
  title={GeoRemover: Removing Objects and Their Causal Visual Artifacts},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2509.18538}
}

• arXiv: 2509.18538