Ultrahr Eval

ultrahr-eval

UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Dataset — Chen Zhao et al. (2025) (arXiv:2510.20661, 2025)

What this evaluates

Evaluates text-to-image diffusion models on ultra-high-resolution generation, probing semantic alignment with prompts, fine-grained texture preservation, and overall perceptual quality at resolutions ≥4096px.

Datasets

• UltraHR-eval4K — total ?; splits: test (-1)
• Aesthetic-Eval@4096 — total 195; splits: test (195)

Metrics

• FID (primary) — range: other
  • Fréchet Inception Distance measures the distance between feature distributions of real and generated images using Inception v3 activations. Lower values indicate better overall image quality and diversity.
• FID_patch — range: other
  • Same as FID but computed on local image patches extracted from the generated and reference images to measure local detail fidelity.
• IS — range: other
  • Inception Score evaluates image quality and diversity by training a classifier on generated images and measuring the KL divergence between conditional and marginal class distributions. Higher values indicate better quality.
• IS_patch — range: other
  • Same as IS but computed on local image patches to evaluate local detail quality and diversity.
• CLIP — range: [0, 1]
  • Long CLIP score computes the cosine similarity between the CLIP text encoder output for the prompt and the CLIP image encoder output for the generated image. Higher values indicate better text-image consistency.
• FG-CLIP — range: [0, 1]
  • Fine-Grained CLIP score uses a specialized CLIP variant or feature extraction method to better capture detailed semantic alignment between prompts and images. Higher values indicate better fine-grained consistency.

Input / output format

Input: Text prompts for generation; reference images for FID/IS computation.

Output: Generated images at 4096×4096 resolution (or short side >4096px for Aesthetic-Eval).

Scoring recipe

def compute_metrics(gen_images, ref_images, prompts):
    fid = frechet_inception_distance(ref_images, gen_images)
    is_score = inception_score(gen_images)
    patches_g = extract_patches(gen_images, size=299)
    patches_r = extract_patches(ref_images, size=299)
    fid_patch = frechet_inception_distance(patches_r, patches_g)
    is_patch = inception_score(patches_g)
    clip_sim = cosine_similarity(clip_encode(prompts), clip_encode(gen_images))
    fg_clip_sim = cosine_similarity(fg_clip_encode(prompts), fg_clip_encode(gen_images))
    return {'FID': fid, 'FID_patch': fid_patch, 'IS': is_score, 'IS_patch': is_patch, 'CLIP': clip_sim, 'FG-CLIP': fg_clip_sim}

Common pitfalls

• Patch-level metrics require consistent tiling/cropping strategies that vary across implementations.
• CLIP/FG-CLIP scores can be misleading if UHR outputs are naively downsampled to 224px before encoding.
• User study used only 5 volunteers, making preference ratings statistically underpowered.

Evidence (verbatim from paper)

We employ several metrics to assess the quality of the generated images, with a particular focus on our evaluation sets, UltraHR-eval4K. To evaluate image-text consistency, we calculate the long CLIP score [zhang2024long] and Fine-Grained (FG) CLIP score [xie2025fg]. Additionally, the Fréchet Inception Distance (FID) [heusel2017gans] and Inception Score (IS) [salimans2016improved] are computed to evaluate the overall image quality of the generated images. Following previous works [bu2025hiflow] ; [renultrapixel], we compute the FID-patch and IS-patch to evaluate the local quality and details of the images, which are based on local image patches.

Citation

@misc{zhao2025ultrahr100k,
  title={UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Dataset},
  author={Chen Zhao et al. (2025)},
  year={2025},
  note={arXiv:2510.20661}
}

• arXiv: 2510.20661