# Cyclegan Eval

> Evaluates the ability of generative models to perform unpaired image-to-image translation while preserving structural integrity and achieving perceptual realism. Probes domain mapping capabilities without requiring paired training data. Use when the user wants to benchmark on Cityscapes, Google Maps aerial photos & maps, or asks about evaluating this task. Reports FCN score.

- Skill: `qhjqhj00/cyclegan-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/cyclegan-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/cyclegan-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/cyclegan-eval

---


# cyclegan-eval

> Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks — Jun-Yan Zhu et al. (2017) (arXiv:1703.10593, 2017)

## What this evaluates

Evaluates the ability of generative models to perform unpaired image-to-image translation while preserving structural integrity and achieving perceptual realism. Probes domain mapping capabilities without requiring paired training data.

## Datasets

- **Cityscapes** — total ?; splits: test (-1)
- **Google Maps aerial photos & maps** — total ?; splits: test (-1)

## Metrics

- `AMT perceptual realism` — range: percent
  - Percentage of Amazon Mechanical Turk participants who incorrectly identify a generated image as real in a forced-choice 'real vs fake' test. Calculated over 40 trials per participant across 25 participants per algorithm.
- `FCN score` **(primary)** — range: [0, 1]
  - Semantic segmentation metrics (per-pixel accuracy, per-class accuracy, mean class Intersection-Over-Union) computed by running an off-the-shelf FCN on the generated photo and comparing the predicted label map against the input ground truth label map.
- `Semantic segmentation metrics` — range: [0, 1]
  - Per-pixel accuracy, per-class accuracy, and mean class Intersection-Over-Union (Class IoU) computed by comparing the generated photo against the ground truth label map using standard Cityscapes benchmark metrics.

## Input / output format

**Input**: 256x256 images from the source domain (e.g., semantic label maps, aerial photos, paintings, or edge maps).

**Output**: 256x256 translated images in the target domain (e.g., photos, maps, artistic styles, or solid images).

## Scoring recipe

```python
def compute_amr_score(generated_images, real_images):
    correct_fake = 0
    for gen, real in zip(generated_images, real_images):
        if participant_clicks(gen):
            correct_fake += 1
    return correct_fake / 40

def compute_fcn_score(generated_image, gt_label_map):
    pred_label_map = FCN.predict(generated_image)
    per_pixel_acc = accuracy(pred_label_map, gt_label_map)
    per_class_acc = mean(class_accuracy(pred_label_map, gt_label_map))
    class_iou = mean(intersection_over_union(pred_label_map, gt_label_map))
    return per_pixel_acc, per_class_acc, class_iou
```

## Common pitfalls

- AMT scores are not directly comparable to the pix2pix paper due to different ground truth processing and participant pools.
- FCN score only evaluates the label-to-photo direction, not the photo-to-label direction.
- Baseline comparisons are only valid within the same experimental setup; cross-paper quantitative comparisons are invalid.

## Evidence (verbatim from paper)

> For this, we adopt the “FCN score” from[[22]], and use it to evaluate the Cityscapes labels$
ightarrow$photo task. The FCN metric evaluates how interpretable the generated photos are according to an off-the-shelf semantic segmentation algorithm (the fully-convolutional network, FCN, from[[33]]). The FCN predicts a label map for a generated photo. This label map can then be compared against the input ground truth labels using standard semantic segmentation metrics described below.

## Citation

```bibtex
@misc{zhu2017cyclegan,
  title={Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks},
  author={Jun-Yan Zhu et al. (2017)},
  year={2017},
  note={arXiv:1703.10593}
}
```

- arXiv: 1703.10593