Epd Eval

epd-eval

Embodied Image Quality Assessment for Robotic Intelligence — Jianbo Zhang et al. (2024) (arXiv:2412.18774, 2024)

What this evaluates

Evaluates how well image quality assessment models correlate with actual robotic task performance under various image distortions. It probes whether traditional human-centric visual quality metrics align with the perception needs of embodied robots performing push and pick tasks.

Datasets

• EPD — total 12500; splits: train (10000), val (2500); repo https://github.com/Jianbo-maker/EPD_benchmark

Metrics

• SRCC — range: [-1, 1]
  • Spearman Rank Correlation Coefficient; measures predictive monotonicity between objective quality scores and ground-truth task performance scores.
• KRCC — range: [-1, 1]
  • Kendall Rank Order Correlation Coefficient; measures predictive monotonicity between predicted and actual scores.
• PLCC (primary) — range: [-1, 1]
  • Pearson Linear Correlation Coefficient; measures predictive accuracy between objective quality scores and ground-truth task performance scores.

Input / output format

Input: 128×128 pixel images captured by a robot-mounted monocular camera during push/pick tasks, including 25 distortion types at 5 intensity levels.

Output: Continuous quality scores predicted by the IQA model, compared against robot task completion scores (average reward) or subjective annotations.

Scoring recipe

def compute_metrics(predictions, gold_scores):
    srcc = spearmanr(predictions, gold_scores).correlation
    krcc = kendalltau(predictions, gold_scores).correlation
    plcc = pearsonr(predictions, gold_scores).correlation
    return {'SRCC': srcc, 'KRCC': krcc, 'PLCC': plcc}

Common pitfalls

• Applying human-centric full-reference IQA methods directly to robotic data, which consistently underperform because robots lack pristine reference images and prioritize task-relevant features over semantic appeal.
• Evaluating only on the aggregate 'All Tasks' split without analyzing subtasks (Push vs. Pick), as distortion sensitivity varies significantly across different robotic manipulation tasks.
• Ignoring the computational constraints of edge deployment; transformer-based models may achieve slightly higher correlation but are impractical for real-time embodied robots compared to lightweight CNNs.

Evidence (verbatim from paper)

Three metrics, SRCC, Kendall Rank Order Correlation Coefficient (KRCC) and PLCC, are used to evaluate the consistency between objective and subjective quality scores. The SRCC and KRCC represent predictive monotonicity and PLCC represents predictive accuracy.

Citation

@misc{zhang2024embodied,
  title={Embodied Image Quality Assessment for Robotic Intelligence},
  author={Jianbo Zhang et al. (2024)},
  year={2024},
  note={arXiv:2412.18774}
}

• arXiv: 2412.18774