Edge LLM Inference Eval

edge_llm_inference_eval

Generative AI on the Edge: Architecture and Performance Evaluation — Nezami et al. (2024) (arXiv:2411.17712, 2024)

What this evaluates

Evaluates the feasibility and performance of deploying various LLMs on CPU-only edge hardware (Raspberry Pi 5 clusters) by measuring inference speed, resource consumption, and reasoning accuracy under constrained conditions.

Datasets

• OpenAssistant/oasst1 (subset) — total 50; splits: test (50); HF OpenAssistant/oasst1
• Winogrande — total ?; splits: test (-1); HF automated-research-group/winogrande

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions on the Winogrande NLI task. The model selects the answer option with the highest log likelihood and compares it to the ground truth label.
• throughput — range: tokens/sec
  • Number of tokens generated per second, measured separately for the Prefill and Decode phases.
• latency — range: ms or seconds
  • Time per token (ms) and total end-to-end generation time (seconds) from input to output completion.
• memory_usage — range: GB
  • Peak RAM consumption during inference, measured in gigabytes.
• cpu_utilization — range: percent
  • Average CPU load percentage across all cores during inference.
• coefficient_of_variation — range: dimensionless
  • Ratio of the standard deviation to the mean throughput across varying context lengths, used to measure performance stability.

Input / output format

Input: Conversational prompts from OASST1 (average 25 words, range 1–241 words). Maximum token generation is capped at 500.

Output: Autoregressive text generation up to 500 tokens. For accuracy evaluation, the model outputs the selected answer option based on highest log likelihood.

Scoring recipe

def evaluate_throughput_latency(prompt, model, max_tokens=500):
    start = time()
    tokens = model.generate(prompt, max_tokens=max_tokens)
    total_time = time() - start
    throughput = len(tokens) / total_time
    latency_per_token = total_time / len(tokens)
    return throughput, latency_per_token

def evaluate_accuracy(dataset, model):
    correct = 0
    for item in dataset:
        log_probs = [model.log_likelihood(item.context, opt) for opt in item.options]
        predicted = item.options[argmax(log_probs)]
        if predicted == item.label:
            correct += 1
    return correct / len(dataset)

def evaluate_stability(throughputs):
    return std(throughputs) / mean(throughputs)

Common pitfalls

• Quantization significantly impacts accuracy and throughput, but the exact bit-width used for each model is not consistently reported across all experiments.
• Prefill and Decode phases exhibit different performance characteristics; evaluating only end-to-end latency masks phase-specific bottlenecks.
• CPU-only edge hardware (Raspberry Pi 5) lacks GPU acceleration, making results non-transferable to GPU-based edge deployments.

Evidence (verbatim from paper)

We assess five key metrics—memory usage, CPU utilization, latency, accuracy, and computational throughput—offering a comprehensive view of resource demands and efficiency for PromptAI in resource-constrained settings.

Citation

@misc{nezami2024edgeai,
  title={Generative AI on the Edge: Architecture and Performance Evaluation},
  author={Nezami et al. (2024)},
  year={2024},
  note={arXiv:2411.17712}
}

• arXiv: 2411.17712