Cavbench Eval

cavbench-eval

CAVBench: A Benchmark Suite for Connected and Autonomous Vehicles — Wang et al. (2018) (arXiv:1810.06659, 2018)

What this evaluates

Evaluates the computational performance and resource efficiency of edge computing platforms for connected and autonomous vehicle workloads. It probes how well hardware handles real-time vision, deep learning, and diagnostic tasks under varying resource constraints.

Datasets

• CAVBench — total ?; splits: SLAM (-1), Object Detection (-1), Object Tracking (-1), Battery Diagnostics (-1), Speech Recognition (-1), Edge Video Analysis (-1)

Metrics

• Matching Factor (MF) (primary) — range: other
  • Calculated as the area under the Quality of Service - Resource Utilization (QoS-RU) curve for each system resource (CPU, memory, bandwidth). QoS is derived from application latency/throughput, and the curve plots QoS against resource allocation levels to indicate platform suitability.
• Average Latency — range: ms
  • Mean execution time per module (for computer vision apps) or per application (for deep learning apps) measured in milliseconds across the dataset.
• Tail Latency — range: ms
  • Maximum or high-percentile (e.g., p95/p99) execution time reported for deep learning applications (SSD, EVBattery, DeepSpeech) to capture worst-case performance.

Input / output format

Input: Real-world datasets (stereo/monocular images, audio recordings, battery log text) processed by six specific CAV applications (ORB-SLAM2, SSD, CIWT, EVBattery, DeepSpeech, OpenALPR).

Output: Per-module or per-application average and tail latency (ms), and QoS-RU curves mapping resource utilization to application QoS.

Scoring recipe

def compute_cavbench_metrics(applications, resource_levels):
    results = {}
    for app in applications:
        latencies = []
        for res in resource_levels:
            latency = run_application(app, res)
            latencies.append(latency)
        avg_lat = mean(latencies)
        tail_lat = max(latencies)
        qos = [1.0 / l for l in latencies]
        mf = trapezoidal_area(qos, [r.utilization for r in resource_levels])
        results[app] = {'avg_latency': avg_lat, 'tail_latency': tail_lat, 'MF': mf}
    return results

Common pitfalls

• Latency reporting differs by workload type: computer vision apps report per-module average latency, while deep learning apps report application-level average and tail latency.
• The Matching Factor is highly dependent on the specific resource utilization curve (CPU, memory, bandwidth), making cross-architecture comparisons sensitive to how QoS is normalized.
• Some datasets (e.g., Mozilla Corpus for speech recognition) lack vehicular background noise, potentially overestimating real-world QoS.

Evidence (verbatim from paper)

We use the area under the curve of each system resource to calculate the Matching Factor (MF) between the application and the platform, indicating whether the platform is suitable for the CAVs application.

Citation

@misc{wang2018cavbench,
  title={CAVBench: A Benchmark Suite for Connected and Autonomous Vehicles},
  author={Wang et al. (2018)},
  year={2018},
  note={arXiv:1810.06659}
}

• arXiv: 1810.06659