Cyclegan Eval

cyclegan-eval

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks — Jun-Yan Zhu et al. (2017) (arXiv:1703.10593, 2017)

What this evaluates

Evaluates the ability of generative models to perform unpaired image-to-image translation while preserving structural integrity and achieving perceptual realism. Probes domain mapping capabilities without requiring paired training data.

Datasets

• Cityscapes — total ?; splits: test (-1)
• Google Maps aerial photos & maps — total ?; splits: test (-1)

Metrics

• AMT perceptual realism — range: percent
  • Percentage of Amazon Mechanical Turk participants who incorrectly identify a generated image as real in a forced-choice 'real vs fake' test. Calculated over 40 trials per participant across 25 participants per algorithm.
• FCN score (primary) — range: [0, 1]
  • Semantic segmentation metrics (per-pixel accuracy, per-class accuracy, mean class Intersection-Over-Union) computed by running an off-the-shelf FCN on the generated photo and comparing the predicted label map against the input ground truth label map.
• Semantic segmentation metrics — range: [0, 1]
  • Per-pixel accuracy, per-class accuracy, and mean class Intersection-Over-Union (Class IoU) computed by comparing the generated photo against the ground truth label map using standard Cityscapes benchmark metrics.

Input / output format

Input: 256x256 images from the source domain (e.g., semantic label maps, aerial photos, paintings, or edge maps).

Output: 256x256 translated images in the target domain (e.g., photos, maps, artistic styles, or solid images).

Scoring recipe

def compute_amr_score(generated_images, real_images):
    correct_fake = 0
    for gen, real in zip(generated_images, real_images):
        if participant_clicks(gen):
            correct_fake += 1
    return correct_fake / 40

def compute_fcn_score(generated_image, gt_label_map):
    pred_label_map = FCN.predict(generated_image)
    per_pixel_acc = accuracy(pred_label_map, gt_label_map)
    per_class_acc = mean(class_accuracy(pred_label_map, gt_label_map))
    class_iou = mean(intersection_over_union(pred_label_map, gt_label_map))
    return per_pixel_acc, per_class_acc, class_iou

Common pitfalls

• AMT scores are not directly comparable to the pix2pix paper due to different ground truth processing and participant pools.
• FCN score only evaluates the label-to-photo direction, not the photo-to-label direction.
• Baseline comparisons are only valid within the same experimental setup; cross-paper quantitative comparisons are invalid.

Evidence (verbatim from paper)

For this, we adopt the “FCN score” from[[22]], and use it to evaluate the Cityscapes labels$ ightarrow$photo task. The FCN metric evaluates how interpretable the generated photos are according to an off-the-shelf semantic segmentation algorithm (the fully-convolutional network, FCN, from[[33]]). The FCN predicts a label map for a generated photo. This label map can then be compared against the input ground truth labels using standard semantic segmentation metrics described below.

Citation

@misc{zhu2017cyclegan,
  title={Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks},
  author={Jun-Yan Zhu et al. (2017)},
  year={2017},
  note={arXiv:1703.10593}
}

• arXiv: 1703.10593