Mlperf Tiny Eval

mlperf-tiny-eval

Co-Design of CNN Accelerators for TinyML using Approximate Matrix Decomposition — Hernández Morales et al. (2026) (arXiv:2604.16113, 2026)

What this evaluates

Evaluates the classification accuracy and hardware efficiency of CNN accelerators on resource-constrained TinyML workloads. It probes the trade-off between inference latency, energy consumption, and model accuracy under post-training approximate matrix decomposition.

Datasets

• MLPerfTiny — total ?; splits: test (-1)

Metrics

• Top-1 Accuracy (primary) — range: percent
  • Percentage of correctly classified samples on the test set, computed using TFLite.
• Latency — range: other
  • Total inference time per model calculated on a layer-wise basis via Vivado simulation.
• Energy Efficiency — range: other
  • Peak Throughput (GOPS) divided by Estimated Power (mW).

Input / output format

Input: Pre-trained CNN models (ResNet, MobileNetV1, DS-CNN) with decomposed weights and 8-bit activations. Inputs are drawn from the MLPerfTiny test set for accuracy evaluation, and uniformly random inputs for functional verification.

Output: Classification labels (for accuracy), layer-wise latency (μs), peak throughput (GOPS), estimated power (mW), estimated energy (μJ), and FPGA resource utilization (LUTs, FFs, BRAMs).

Scoring recipe

def compute_top1_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_latency_and_energy(layer_latencies, toggle_rates, resource_model, peak_throughput_gops):
    latency_us = sum(layer_latencies)
    power_mw = estimate_power(toggle_rates, resource_model)
    energy_uj = (power_mw * latency_us) / 1000
    energy_efficiency = peak_throughput_gops / power_mw
    return latency_us, energy_uj, energy_efficiency

Common pitfalls

• Uses a 10% test set split for genetic algorithm exploration and 90% for final accuracy reporting, which differs from standard full-test-set evaluation.
• Accuracy is evaluated using TFLite on the host CPU, while hardware metrics (latency, power) are derived from FPGA synthesis/simulation, creating a potential host-hardware evaluation gap.
• Normalized speedup is calculated against an 8-bit baseline systolic array, not the original floating-point model, which can obscure absolute latency baselines.

Evidence (verbatim from paper)

The 4-bit SA, although providing advantages in many hardware performance metrics, causes a top-1 accuracy drop of at least 6% for the analyzed CNNs, which is why this configuration is excluded in the following comparisons. TFLite is used to evaluate the accuracy of both our decomposed and the baseline CNNs. The final latency is calculated for each CNN on a layer-wise basis using Vivado simulation, from which toggle rates are also obtained to serve as inputs for the power consumption analysis.

Citation

@misc{hernandezmorales2026codesign,
  title={Co-Design of CNN Accelerators for TinyML using Approximate Matrix Decomposition},
  author={Hernández Morales et al. (2026)},
  year={2026},
  note={arXiv:2604.16113}
}

• arXiv: 2604.16113