Controllable Gen Eval

controllable-gen-eval

OmniGen: Unified Image Generation — Shitao Xiao et al. (2024) (arXiv:2409.11340, 2024)

What this evaluates

Evaluates controllable image generation based on visual conditions (segmentation masks, edges, depth maps) by measuring how closely the generated image's extracted conditions match the input conditions. It tests spatial and structural controllability.

Datasets

• ControlNet++ dataset — total ?; splits: test (-1)

Metrics

• mIoU (Seg. Mask) (primary) — range: [0, 1]
  • Mean Intersection over Union between the input segmentation mask and the mask extracted from the generated image.
• F1 Score (Canny Edge) — range: [0, 1]
  • F1 score between the input Canny edge map and the edge map extracted from the generated image.
• SSIM (Hed Edge) — range: [0, 1]
  • Structural Similarity Index Measure between the input HED edge map and the extracted edge map.
• RMSE (Depth Map) — range: [0, inf)
  • Root Mean Square Error between the input depth map and the depth map extracted from the generated image.

Input / output format

Input: Reference image (or condition map) and text instruction.

Output: Generated RGB image.

Scoring recipe

for condition in conditions:
  gen_img = model.generate(condition, instruction)
  extracted_cond = extract_condition(gen_img)
  if condition == 'mask': score = compute_miou(condition, extracted_cond)
  elif condition == 'canny': score = compute_f1(condition, extracted_cond)
  elif condition == 'hed': score = compute_ssim(condition, extracted_cond)
  elif condition == 'depth': score = compute_rmse(condition, extracted_cond)
  scores.append(score)

Common pitfalls

• Controllability is evaluated by extracting conditions from the generated image, which introduces detector/model bias.
• RMSE for depth is lower-is-better, while others are higher-is-better, requiring careful aggregation.

Evidence (verbatim from paper)

In Table[4], we use the dataset and script from[[31]] to evaluate the generation capability based on visual conditions. For each condition, the controllability is evaluated by measuring the similarity between the input conditions and the extracted conditions from generated images of diffusion models.

Citation

@misc{xiao2024omnigen,
  title={OmniGen: Unified Image Generation},
  author={Shitao Xiao et al. (2024)},
  year={2024},
  note={arXiv:2409.11340}
}

• arXiv: 2409.11340