qhjqhj00 visual-quality-assessment-eval
Technical Report on Visual Quality Assessment for Frame Interpolation — Men et al. (2019) (arXiv:1901.05362, 2019)
What this evaluates
Evaluates the perceptual visual quality of interpolated frames generated by optical flow methods against human judgments. It measures how well traditional objective metrics like RMSE correlate with crowdsourced subjective quality ratings across multiple video sequences.
Datasets
- Middlebury — total ?; splits: test (-1)
Metrics
SROCC(primary) — range: [-1, 1]- Spearman Rank Order Correlation Coefficient measuring the monotonic relationship between RMSE-based rankings and subjective quality rankings. Computed with 95% confidence intervals via bootstrapping (1000 iterations) and Fisher transform.
subjective quality score— range: [0, 1]- Scale values reconstructed from pairwise comparisons using Thurstone's model, linearly rescaled to [0, 1] by anchoring the worst-quality image to 0 and the ground-truth image to 1.
Input / output format
Input: Pairwise comparisons of interpolated frames from 141 optical flow methods across 8 Middlebury sequences, presented to crowd workers alongside two anchor images (worst quality and ground truth).
Output: Binary preference choices per pair, aggregated into rescaled subjective quality scores in [0, 1].
Scoring recipe
def compute_protocol(prefs, rmse_values):
# Reconstruct scores via Thurstone's model with anchors
raw_scores = thurstone_reconstruct(prefs, anchors=[0, 1])
# Rescale to [0, 1]
subjective_scores = rescale_to_zero_one(raw_scores)
# Compute Spearman rank correlation
srocc = spearman_rank_correlation(rmse_values, subjective_scores)
# Bootstrap confidence intervals
ci_95 = bootstrap_ci(srocc, n_iterations=1000)
return srocc, ci_95, subjective_scores
Common pitfalls
- RMSE correlates poorly with human perception for frame interpolation due to artifacts not captured by pixel-wise error.
- Subjective scores require strict anchor calibration (worst/ground-truth) to ensure comparability across separate crowdsourcing jobs.
- Crowd workers must pass accuracy filters (≥70% on test questions) to be included in the final trusted worker pool.
Evidence (verbatim from paper)
Table II shows the bootstrapped (after 1000 iterations) SROCC correlation values accompanied with confidence intervals (95%) between the ranking in Middlebury benchmark (i.e., ranking according to RMSE) and the re-ranking according to our subjective study.
Citation
@misc{men2019visual,
title={Technical Report on Visual Quality Assessment for Frame Interpolation},
author={Men et al. (2019)},
year={2019},
note={arXiv:1901.05362}
}
- arXiv: 1901.05362