Chd Cmms

chd-cmms

Evaluating Generative Models via One-Dimensional Code Distributions — Jia et al. (2026) (arXiv:2603.08064, 2026)

What this evaluates

Evaluates the perceptual quality and human preference alignment of generative image models by comparing their discrete visual token distributions and sequence statistics against human ratings.

Datasets

• AGIQA — total 2982; splits: test (-1)
• HPDv2 — total 430060; splits: test (-1)
• HPDv3 — total 1080000; splits: test (-1)
• VisForm — total ?; splits: test (-1)

Metrics

• CHD (primary) — range: other
  • Codebook Histogram Distance. Computes the Hellinger distance between the 1D unigram and 2D co-occurrence histograms of discrete visual tokens generated by the model versus a reference distribution.
• CMMS (primary) — range: [0, 1]
  • Code Mixture Model Score. A Transformer-MLP regressor trained on ImageNet-1K token sequences to predict perceptual quality, calibrated via an exponential mapping exp(-20p). Higher is better.
• Spearman's rank correlation — range: [-1, 1]
  • Measures rank-order consistency between predicted scores and human ratings.
• Kendall's tau — range: [-1, 1]
  • Measures pairwise consistency between predicted scores and human ratings.
• N-MSE — range: [0, 1]
  • Normalized Mean Squared Error capturing the normalized deviation between predicted scores and human ratings.
• Pairwise accuracy — range: [0, 1]
  • Fraction of image pairs where the metric selects the same winner as human judges.

Input / output format

Input: Generated images (for CHD) or their discrete 1D visual token sequences (for CMMS).

Output: CHD outputs a scalar distance score (lower is better). CMMS outputs a scalar quality score (higher is better). Correlation metrics output a scalar.

Scoring recipe

def compute_chd(gen_images, ref_images, tokenizer):
    gen_tokens = tokenizer.encode(gen_images)
    ref_tokens = tokenizer.encode(ref_images)
    gen_hist = compute_histograms(gen_tokens) # 1D unigram + 2D co-occurrence
    ref_hist = compute_histograms(ref_tokens)
    return hellinger_distance(gen_hist, ref_hist)

def compute_cmms(gen_images, cmms_model, tokenizer):
    tokens = tokenizer.encode(gen_images)
    raw_scores = cmms_model.predict(tokens)
    return np.exp(-20 * raw_scores) # Exponential calibration

def compute_correlation(pred_scores, human_ratings):
    return spearmanr(pred_scores, human_ratings), kendalltau(pred_scores, human_ratings), nmse(pred_scores, human_ratings)

Common pitfalls

• CHD requires a reference distribution of real images to compute histogram distances; it is not a purely no-reference metric.
• CMMS must be trained once on ImageNet-1K and applied to all benchmarks without fine-tuning; fine-tuning violates the proposed protocol.
• Token sequence length and input resolution significantly impact performance; using 128 tokens at 256x256 resolution is optimal per the ablation study.

Evidence (verbatim from paper)

We quantify agreement between objective metrics and human judgments using Spearman’s rank correlation, Kendall’s tau, and normalized mean squared error (N-MSE). Spearman and Kendall measure rank- and pairwise-level consistency, respectively, while N-MSE captures normalized deviation between predicted scores and human ratings. For preference prediction, we additionally report pairwise accuracy: the fraction of image pairs for which the metric selects the same winner as humans.

Citation

@misc{jia2026chdcmms,
  title={Evaluating Generative Models via One-Dimensional Code Distributions},
  author={Jia et al. (2026)},
  year={2026},
  note={arXiv:2603.08064}
}

• arXiv: 2603.08064