Edge LLM Inference Benchmark

edge-llm-inference-benchmark

Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers — Renney et al. (2026) (arXiv:2604.24785, 2026)

What this evaluates

Evaluates the trade-offs between token throughput, latency, energy efficiency, and physical footprint when deploying compact LLMs on various IoT-grade single-board computers with different hardware accelerators (CPU, NPU, GPU).

Datasets

• Static Prompt Benchmark — total 1; splits: test (1); repo https://github.com/SquidyBallinx11011/LLM-Edge-Benchmarking-Suite

Metrics

• Throughput (tokens/s) (primary) — range: tokens/s
  • Total tokens generated divided by the total elapsed wall-clock time for the generation run.
• Time-to-first-token (TTFT) (primary) — range: seconds
  • Wall-clock time elapsed between prompt submission and the receipt of the first generated token.
• Energy per million tokens (MJ/Mtok) (primary) — range: MJ/Mtok
  • Total energy consumed (derived from power meter readings and generation time) scaled to represent the energy required to generate one million tokens.
• Throughput density (Tps/m3) — range: Tps/m3
  • Token throughput per second normalised by the physical device volume in cubic metres, quantifying inference capability per unit of space.

Input / output format

Input: Fixed prompt: "Explain why the sky is blue in two or more paragraphs." with generation length capped at 100 tokens via num_predict.

Output: Streaming text generation, up to 100 tokens, returned incrementally as they are generated.

Scoring recipe

def compute_metrics(wall_start, first_token_time, wall_end, tokens_count, power_watts, device_volume_m3):
    total_time = wall_end - wall_start
    throughput = tokens_count / total_time
    ttft = first_token_time - wall_start
    energy_joules = power_watts * total_time
    energy_per_mtok = (energy_joules / 1_000_000) / (tokens_count / 1_000_000)
    throughput_density = throughput / device_volume_m3
    return throughput, ttft, energy_per_mtok, throughput_density

# Average results over n=5 runs after a warmup request

Common pitfalls

• Failing to issue a warmup request before timed runs, which skews initial throughput and TTFT due to model loading overhead.
• Comparing raw energy consumption without scaling to a fixed token count (e.g., 1M tokens), which unfairly penalizes slower hardware or shorter generation runs.
• Ignoring quantization precision differences across runtimes (e.g., Q4_K_M on CPU vs INT8/INT4 on NPUs/GPUs), which significantly alters accelerator utilization and power draw.

Evidence (verbatim from paper)

Performance was evaluated using three primary metrics: Throughput (tokens/s): the rate of token generation during inference. Time-to-first-token (TTFT): latency between prompt submission and generation of the first token. Energy consumption (MJ/Mtok): energy required to generate one million tokens, capturing hardware efficiency.

Citation

@misc{renney2026cloudtoedge,
  title={Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers},
  author={Renney et al. (2026)},
  year={2026},
  note={arXiv:2604.24785}
}

• arXiv: 2604.24785