Pico Tinyml Benchmark Eval

pico-tinyml-benchmark-eval

Real-Time Performance Benchmarking of TinyML Models in Embedded Systems (PICO: Performance of Inference, CPU, and Operations) — Dey et al. (2025) (arXiv:2509.04721, 2025)

What this evaluates

Evaluates real-time performance and resource efficiency of TinyML models on embedded hardware by measuring inference latency, CPU/memory utilization, and prediction confidence across different platforms.

Datasets

• Gesture Classification — total ?; splits: test (-1)
• Keyword Spotting — total ?; splits: test (-1)
• MobileNet V2 — total ?; splits: test (-1)

Metrics

• Average Inference Latency (ms) (primary) — range: other
  • Average time in milliseconds to process a single inference request across all test samples.
• CPU Utilization (%) — range: percent
  • Percentage of CPU capacity consumed during model inference.
• Memory Utilization (%) — range: percent
  • Percentage of available RAM consumed during model inference.
• Prediction Confidence Score — range: other
  • Average model output confidence/probability across test iterations.

Input / output format

Input: Raw sensor or image/audio inputs corresponding to the three tasks, processed by the respective TinyML model on the target embedded platform.

Output: Model predictions alongside system telemetry: inference latency (ms), CPU usage (%), memory usage (%), and prediction confidence score.

Scoring recipe

def compute_metrics(predictions, gold, telemetry):
    latencies = [t['latency_ms'] for t in telemetry]
    cpu_usages = [t['cpu_pct'] for t in telemetry]
    mem_usages = [t['mem_pct'] for t in telemetry]
    confidences = [t['confidence'] for t in telemetry]
    return {
        'avg_latency_ms': sum(latencies) / len(latencies),
        'avg_cpu_pct': sum(cpu_usages) / len(cpu_usages),
        'avg_mem_pct': sum(mem_usages) / len(mem_usages),
        'avg_confidence': sum(confidences) / len(confidences)
    }

Common pitfalls

• Latency is reported only as an average, omitting tail latency or variance which are critical for real-time guarantees.
• Resource utilization metrics capture peak or average usage during inference but exclude background OS overhead and framework initialization costs.
• Confidence scores for MobileNet V2 are reported as ~17.16, indicating a non-standard scaling or aggregation method that differs from typical probability outputs.

Evidence (verbatim from paper)

Table I presents the average inference latency for all models, while Figure 2 illustrates the distribution across platforms. Across all models, the Raspberry Pi 4 consistently outperformed the BeagleBone AI64 in latency.

Citation

@misc{dey2025pico,
  title={Real-Time Performance Benchmarking of TinyML Models in Embedded Systems (PICO: Performance of Inference, CPU, and Operations)},
  author={Dey et al. (2025)},
  year={2025},
  note={arXiv:2509.04721}
}

• arXiv: 2509.04721