Layout To Image Eval

layout-to-image-eval

HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation — Bo Cheng et al. (arXiv:2410.14324, 2024)

What this evaluates

Evaluates a model's ability to generate high-fidelity images conditioned on spatial layouts and text descriptions, measuring both perceptual quality and precise object-level spatial alignment.

Datasets

• COCO-3K — total 3000; splits: test (3000)
• HiCo-7K — total 7000; splits: test (7000)

Metrics

• FID (primary) — range: other
  • Frechet Inception Distance measuring the distance between feature distributions of real and generated images. Lower values indicate higher perceptual quality.
• IS — range: other
  • Inception Score combining image quality and diversity using a pretrained Inception network. Higher values indicate better perceptual quality.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity measuring perceptual difference between images using deep features. Lower values indicate higher similarity.
• YOLO AP/AR — range: [0, 100]
  • Average Precision and Average Recall computed using a pretrained YOLOv4 detector on generated images. Higher values indicate better object recognizability and detection recall.
• Local CLIP Score — range: other
  • Cosine similarity between local CLIP embeddings of detected object regions and their corresponding text descriptions. Higher values indicate better text-image consistency.
• Local IoU Score — range: [0, 100]
  • Average maximum Intersection-over-Union between detected object bounding boxes and ground truth layout boxes. Higher values indicate better spatial alignment.

Input / output format

Input: Layout bounding boxes/regions and corresponding text descriptions/captions.

Output: Generated RGB images (typically 256×256 or 512×512 resolution).

Scoring recipe

# Spatial evaluation on HiCo-7K
# 1. Detect objects in generated image
detections = GroundingDINO.predict(generated_image)
gt_boxes = layout_annotations

# 2. Filter correct detections based on IoU and CLIP threshold
correct_detections = []
for gt in gt_boxes:
    max_iou = max(iou(det, gt) for det in detections)
    clip_sim = local_clip_similarity(det, gt)
    if max_iou > 0.5 and clip_sim > 0.2:
        correct_detections.append(det)

# 3. Compute detection metrics
ap, ap50, ap75, ar = compute_detection_scores(correct_detections, gt_boxes)

Common pitfalls

• Resolution varies between 256×256 (COCO-3K) and 512×512 (HiCo-7K), which directly impacts FID/IS scores and complicates cross-dataset comparison.
• Zero-shot evaluation on COCO-3K suffers from distribution shift because the model is trained on fine-grained GRIT data, leading to suboptimal controllability on coarse-grained categories.
• Local IoU/CLIP thresholds (0.5 and 0.2) are fixed and may not generalize across different object scales or text complexity without recalibration.

Evidence (verbatim from paper)

Furthermore, we use Grounding-DINO*[[16]]* to detect the instance caption and calculate the maximum IoU between the detection boxes and the ground truth box. If the maximum IoU is higher than the threshod 0.5 and the Local CLIP Score*[[49]]* of them is higher than 0.2, we mark it as position correctly generated. We use AR, AP, AP50 and AP75 to evaluate the spatial performance.

Citation

@misc{cheng2024hico,
  title={HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation},
  author={Bo Cheng et al.},
  year={2024},
  note={arXiv:2410.14324}
}

• arXiv: 2410.14324