Live Meta Mcg Eval

live-meta-mcg-eval

Study of Subjective and Objective Quality Assessment of Mobile Cloud Gaming Videos — Saha et al. (2023) (arXiv:2305.17260, 2023)

What this evaluates

Evaluates the ability of objective video quality assessment models to predict human-perceived quality of mobile cloud gaming videos distorted by compression and resizing artifacts. It benchmarks both general-purpose and gaming-specific no-reference models against human subjective ratings.

Datasets

• LIVE-Meta Mobile Cloud Gaming (LIVE-Meta MCG) — total 600; splits: train_val (-1), test (-1); repo https://github.com/avinabsaha/LIVE-Meta-MCG-SubjectiveStudySetup

Metrics

• SROCC (primary) — range: other
  • Spearman's rank correlation coefficient measuring the monotonic relationship between predicted and human quality scores.
• PLCC — range: other
  • Pearson's linear correlation coefficient computed after fitting a 4-parameter logistic regression to linearize predictions: f(x) = β₂ + (β₁ - β₂) / (1 + exp(-(x + β₃/|β₄|))).
• KRCC — range: other
  • Kendall's rank correlation coefficient measuring the ordinal association between predicted and human quality scores.
• RMSE — range: other
  • Root mean square error of the difference between linearized predictions and human quality scores.

Input / output format

Input: Video sequences (typically 360x720, upscaled to 1080x2160) from mobile cloud gaming sessions.

Output: Predicted video quality score (DMOS/MLE-MOS).

Scoring recipe

def compute_metrics(predictions, gold_mos):
    # Logistic regression to linearize predictions for PLCC/RMSE
    beta = fit_logistic(predictions, gold_mos)
    linear_preds = beta[2] + (beta[0] - beta[2]) / (1 + exp(-(predictions + beta[3]/abs(beta[4]))))
    srocc = spearmanr(predictions, gold_mos)
    krcc = kendalltau(predictions, gold_mos)
    plcc = pearsonr(linear_preds, gold_mos)
    rmse = sqrt(mean((linear_preds - gold_mos)**2))
    return srocc, krcc, plcc, rmse

# Repeat over 1000 random 80/20 content-disjoint splits
# Report median and standard deviation across splits

Common pitfalls

• Models are evaluated over 1000 random content-disjoint train-test splits, not a single fixed split.
• PLCC and RMSE require fitting a 4-parameter logistic regression to linearize predictions before computing correlation/error.
• NIQE is evaluated unsupervised, while others use SVR or deep learning backbones trained on the dataset.

Evidence (verbatim from paper)

We evaluated the performance of the objective NR-VQA algorithms using the following metrics: Spearman’s Rank Order Correlation Coefficient (SROCC), Kendall Rank Correlation Coefficient (KRCC), Pearson’s Linear Correlation Coefficient (PLCC), and Root Mean Square Error (RMSE). The metrics SROCC and KRCC measure the monotonicity of the objective model prediction with respect to human scores, while the metrics PLCC and RMSE measure prediction accuracy. ... We tested the algorithms mentioned above on 1000 random train-test splits using the four metrics.

Citation

@misc{saha2023live_meta_mcg,
  title={Study of Subjective and Objective Quality Assessment of Mobile Cloud Gaming Videos},
  author={Saha et al. (2023)},
  year={2023},
  note={arXiv:2305.17260}
}

• arXiv: 2305.17260