Layoutbench Eval

layoutbench-eval

Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation — Cho et al. (2023) (arXiv:2304.06671, 2023)

What this evaluates

Evaluates layout-guided image generation models on their ability to follow spatial control instructions (number, position, size, shape) across in-distribution and out-of-distribution layouts. Probes generalization to arbitrary object configurations and fine-grained spatial reasoning.

Datasets

• CLEVR — total 70000; splits: train (70000), val (-1)
• LayoutBench — total ?; splits: Number-few (-1), Number-many (-1), Position-center (-1), Position-boundary (-1), Size-tiny (-1), Size-large (-1), Shape-horizontal (-1), Shape-vertical (-1)

Metrics

• AP (AP50) (primary) — range: [0, 1]
  • Average Precision at IoU threshold 0.5, computed using a DETR-R101-DC5 object detector to compare predicted bounding boxes and class labels against ground-truth layout specifications. Higher values indicate better layout adherence.
• FID — range: other
  • Fréchet Inception Distance measuring the feature distribution distance between generated images and ground-truth images. Lower values indicate higher image quality.
• SceneFID — range: other
  • Scene-aware FID measuring the feature distribution distance between bounding boxes in generated images and ground-truth boxes. Lower values indicate better spatial fidelity.

Input / output format

Input: Text-encoded layout specification containing quantized bounding box coordinates (1000 bins) concatenated with object class tokens or textual descriptions, fed into a Stable Diffusion-based generator.

Output: 512x512 RGB image.

Scoring recipe

def compute_metrics(generated_images, gold_layouts):
    # Layout Accuracy (AP50)
    preds = DETR_R101_DC5.predict(generated_images)
    ap50 = compute_AP(preds, gold_layouts, iou_threshold=0.5)
    
    # Image Quality
    fid = compute_FID(generated_images, ground_truth_images)
    scenefid = compute_FID(extract_boxes(generated_images), extract_boxes(ground_truth_images))
    
    return {'AP50': ap50, 'FID': fid, 'SceneFID': scenefid}

Common pitfalls

• Relying solely on FID/SceneFID for evaluation, as models can achieve low scores while completely failing to follow layout constraints.
• Using original CLEVR bounding box annotations without correction, which contain minor errors that artificially lower AP on ground-truth images.
• Assuming models generalize equally across all spatial skills; performance drops significantly on OOD splits like 'many', 'tiny', 'center/boundary', and 'horizontal/vertical' shapes.

Evidence (verbatim from paper)

For quantitative evaluation, we measure layout accuracy and image quality. Layout accuracy is measured by AP (average precision) based on DETR-R101-DC5, as mentioned in [Sec. 3.2]. Higher AP indicates that the generated images follow the input layouts more closely. FID and SceneFID are adopted to measure image quality. Lower FID (SceneFID) indicates that the generated images (boxes) have a more similar feature distribution to the ground-truth ones.

Citation

@misc{cho2023diagnostic,
  title={Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation},
  author={Cho et al. (2023)},
  year={2023},
  note={arXiv:2304.06671}
}

• arXiv: 2304.06671