Xrbench Eval

xrbench-eval

XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse — Kwon et al. (2022) (arXiv:2211.08675, 2022)

What this evaluates

Evaluates ML inference accelerators under realistic Extended Reality (XR) workloads. It probes the system's ability to handle real-time, multi-task, multi-model (MTMM) pipelines with dynamic dependencies while meeting strict latency, energy, and quality-of-experience (QoE) constraints.

Datasets

• XRBench Scenarios — total ?; splits: test (-1)

Metrics

• overall score (primary) — range: [0, 1]
  • Aggregates normalized real-time score (deadline violation), QoE score (frame drop rate), and energy score. The exact weighting formula is not provided in the excerpt, but the metric is designed so that poor performance in any component (e.g., zero real-time score) drastically reduces the final value.
• real-time score — range: [0, 1]
  • Quantifies the degree of deadline violation. Higher-is-better.
• QoE score — range: [0, 1]
  • Captures frame drop rates to reflect user quality-of-experience.
• energy score — range: [0, 1]
  • Captures energy consumption relative to the most efficient design.

Input / output format

Input: Workload scenario profiles specifying model dependency chains (cascaded, concurrent, or cascaded-concurrent), target frame rates, and trigger probabilities for dynamic model execution.

Output: Per-scenario performance scores: real-time score, QoE score, energy score, and overall XRBENCH SCORE, reported as normalized values between 0 and 1.

Scoring recipe

# 1. Compute component scores (normalized 0-1)
realtime_score = compute_deadline_violation_rate(predictions, gold)
qoe_score = 1.0 - compute_frame_drop_rate(predictions, gold)
energy_score = compute_energy_efficiency(predictions, gold)
# 2. Aggregate into overall XRBENCH SCORE
overall_score = aggregate(realtime_score, qoe_score, energy_score)
# Note: The paper emphasizes that overall score penalizes systems that
# meet deadlines but suffer high frame drops or energy waste.

Common pitfalls

• Relying solely on hardware utilization, which ignores frame drops and deadline violations.
• Assuming a high real-time score guarantees good system performance, as it may mask high energy consumption or frame drops.
• Ignoring dynamic cascading probabilities, which significantly alter workload characteristics and accelerator preferences.

Evidence (verbatim from paper)

The real-time score quantifies the degree of deadline violation. Higher-is-better for the real-time score; however, a high real-time score itself does not guarantee ideal system performance. For example, accelerator A with 8K PEs running the Outdoor Activity B (Figure 5, (d)) has a real-time score of 1.0, which indicates that most of the deadlines are met within a small margin. However, accelerator A misses 10.0% of the frames (not shown) and has high energy consumption, 34.1% greater than the most energy-efficient design (accelerator C). Our scoring metric incorporates all aspects, including QoE score for frame drops and energy score for energy consumption, and it reports an overall score of 0.49, which is 42.9% less than the best accelerator (I).

Citation

@misc{kwon2022xrbench,
  title={XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse},
  author={Kwon et al. (2022)},
  year={2022},
  note={arXiv:2211.08675}
}

• arXiv: 2211.08675