Cgiqa 6k Eval

cgiqa-6k-eval

Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Images — Zhang et al. (2023) (arXiv:2303.08050, 2023)

What this evaluates

Evaluates the ability of no-reference image quality assessment (IQA) models to predict human-perceived quality scores for in-the-wild computer graphics images. It probes how well models capture both distortion artifacts and aesthetic quality in synthetic visual content compared to natural scenes.

Datasets

• CGIQA-6k — total 6000; splits: train (-1), test (-1); repo https://github.com/zzc-1998/CGIQA6K
• CCT-CGI — total 528; splits: test (-1)
• NBU-CIQAD — total 2600; splits: test (-1)
• LIVE-YT-Gaming — total 3501; splits: test (-1)

Metrics

• SRCC (primary) — range: other
  • Spearman Rank Correlation Coefficient measuring the similarity between predicted and ground-truth rankings. Computed after non-linear score mapping.
• PLCC — range: other
  • Pearson Linear Correlation Coefficient describing the linear correlation between predicted and ground-truth scores after mapping.
• KRCC — range: other
  • Kendall's Rank Correlation Coefficient denoting the ordinal association between predicted and ground-truth scores.
• RMSE — range: other
  • Root Mean Squared Error standing for the average distance between predicted and ground-truth scores.

Input / output format

Input: 224×224 RGB images (randomly cropped from 256×256 resized inputs)

Output: Continuous predicted quality score per image

Scoring recipe

# 1. Generate predicted scores y for each image
# 2. Fit 5-parameter logistic mapping to y against MOS:
#    y_hat = beta1*(0.5 - 1/(1+exp(beta2*(y-beta3)))) + beta4*y + beta5
# 3. Rescale MOS to 5-point scale
# 4. Compute metrics:
#    SRCC = spearmanr(y_hat, mos)[0]
#    PLCC = pearsonr(y_hat, mos)[0]
#    KRCC = kendalltau(y_hat, mos)[0]
#    RMSE = sqrt(mean((y_hat - mos)**2))

Common pitfalls

• Must use No-Reference (NR) models only, as the database lacks pristine reference images.
• Predicted scores require non-linear 5-parameter logistic mapping before correlation metrics are computed.
• Performance is averaged over 10 random 8:2 train/test splits to mitigate partition variance.

Evidence (verbatim from paper)

Four mainstream consistency evaluation criteria are used to measure the correlation between the predicted scores and MOS, which include Spearman Rank Correlation Coefficient (SRCC), Pearson Linear Correlation Coefficient (PLCC), Kendall's Rank Correlation Coefficient (KRCC), Root Mean Squared Error (RMSE). The SRCC values represent the similarity between two groups of rankings, the PLCC values describe the linear correlation of two sets of rankings, the KRCC values denote the ordinal association between two measured quantities, and the RMSE values stand for the average distance between the predicted scores and labels. Before obtaining the criteria values, a five-parameter logistic function is applied to map the predicted scores according to the practices in [64]: y_hat = beta1*(0.5 - 1/(1 + exp(beta2*(y - beta3)))) + beta4*y + beta5. Besides, the MOSs for all databases are rescaled to a five-point scale for validation.

Citation

@misc{zhang2023cgiqa,
  title={Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Images},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2303.08050}
}

• arXiv: 2303.08050