Miqa Eval

miqa-eval

Image Quality Assessment for Machines: Paradigm, Large-scale Database, and Models — Wang et al. (2025) (arXiv:2508.19850, 2025)

What this evaluates

Evaluates how image degradations impact machine vision system (MVS) performance rather than human perception. It measures the correlation between predicted image quality scores and ground-truth machine task metrics (accuracy and consistency) across classification, detection, and segmentation tasks.

Datasets

• MIQD-2.5M — total 2500000; splits: train (-1), val (-1); repo https://github.com/XiaoqiWang/MIQA

Metrics

• SRCC (primary) — range: [0, 1]
  • Spearman rank correlation coefficient measuring monotonic relationship between predicted scores and ground-truth machine performance scores.
• PLCC — range: [0, 1]
  • Pearson linear correlation coefficient measuring linear relationship between predicted scores and ground-truth machine performance scores.
• KRCC — range: [0, 1]
  • Kendall’s rank correlation coefficient measuring ordinal association between predicted scores and ground-truth machine performance scores.
• RMSE — range: other
  • Root mean square error quantifying the average magnitude of prediction errors relative to ground-truth scores.

Input / output format

Input: Degraded image passed through the MIQA model.

Output: A single continuous predicted quality score.

Scoring recipe

def compute_metrics(pred_scores, gold_scores):
    # Align predictions to ground-truth via 5-parameter logistic regression on validation set
    aligned_pred = nonlinear_regression_align(pred_scores, gold_scores)
    srcc = spearmanr(aligned_pred, gold_scores).correlation
    plcc = pearsonr(aligned_pred, gold_scores).correlation
    krcc = kendallr(aligned_pred, gold_scores).correlation
    rmse = sqrt(mean_squared_error(gold_scores, aligned_pred))
    return {'SRCC': srcc, 'PLCC': plcc, 'KRCC': krcc, 'RMSE': rmse}

Common pitfalls

• Applying HVS-based metrics (PSNR, SSIM) directly without the required nonlinear regression alignment to ground-truth machine performance scores.
• Failing to use stratified sampling for class imbalance, leading to content leakage or skewed validation distributions.
• Confusing 'consistency scores' (robustness to degradation) with 'accuracy scores' (task performance drop) when interpreting results.

Evidence (verbatim from paper)

The performance was evaluated using multiple correlation metrics: SRCC, PLCC, and Kendall’s Rank Correlation Coefficient (KRCC), all ranging from 0 to 1 with higher values indicating better performance, alongside RMSE to quantify prediction accuracy across diverse measurement methods.

Citation

@misc{wang2025miqa,
  title={Image Quality Assessment for Machines: Paradigm, Large-scale Database, and Models},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2508.19850}
}

• arXiv: 2508.19850