LLM Inference Profiling

llm-inference-profiling

ELANA: A Simple Energy and Latency Analyzer for LLMs — Chiang et al. (2025) (arXiv:2512.09946, 2025)

What this evaluates

Evaluates LLM inference efficiency by measuring prefilling latency (TTFT), decoding latency per token (TPOT), end-to-end latency (TTLT), and corresponding energy consumption (J/Prompt, J/Token, J/Request) across varying prompt lengths, batch sizes, and hardware platforms.

Datasets

• (no dataset; pure metric skill)

Metrics

• TTFT (primary) — range: ms
  • Measures the latency of the prefilling stage (processing the entire input prompt before the first output token). Reported in milliseconds (ms).
• TPOT — range: ms
  • Captures the average decoding latency per generated token during autoregressive generation. Computed by recording inter-token generation intervals and averaging them. Reported in ms.
• TTLT — range: ms
  • Measures the complete end-to-end inference latency from receiving the input prompt to generating the final output token. Reported in ms.
• J/Prompt — range: J
  • Energy consumption in Joules during the prefilling stage, calculated by multiplying the average power draw over the measurement window by the TTFT.
• J/Token — range: J
  • Energy consumption in Joules per generated token during decoding, calculated by multiplying the average power draw by the TPOT.
• J/Request — range: J
  • Total energy consumption in Joules for the entire end-to-end request, calculated by multiplying the average power draw by the TTLT.

Input / output format

Input: Random input prompts with user-specified prefilling length ($T_p$) and generation length ($T_g$), processed in batches of size $b_{size}$ across $n_{GPU}$ devices.

Output: Average latency (ms) and energy (J) metrics reported per workload configuration, averaged over 100 runs (or 20 for TTLT/J/Request).

Scoring recipe

power_samples = query_power_sensor(interval=0.1)
avg_power = mean(power_samples)

ttft = measure_time(start_prompt, end_first_token)
tpot = measure_inter_token_intervals()
ttlt = measure_time(start_prompt, end_last_token)

j_prompt = avg_power * ttft
j_token = avg_power * tpot
j_request = avg_power * ttlt

ttft_avg = mean([ttft_i for i in range(100)])
tpot_avg = mean([tpot_i for i in range(100)])
ttlt_avg = mean([ttlt_i for i in range(20)])

Common pitfalls

• Do not cache CUDA graphs for the prefilling stage (TTFT), but do cache them for the generation stage (TPOT) to maximize throughput.
• Energy is computed by multiplying average power (sampled every 0.1s) by latency, not by integrating instantaneous power over time.
• Model/cache sizes use SI units (1 GB = 1000^3 bytes) by default, though binary units (GiB) are optionally supported.

Evidence (verbatim from paper)

TTFT measures the latency of the prefilling (i.e., prompting) stage, where the entire input prompt is processed before the model generates the first output token. This metric reflects the latency of the initial forward pass, and is particularly important for interactive applications such as chat assistants or long-context summarization. ELANA provides accurate TTFT measurements by isolating the prefilling stage and reporting both raw latency and averaged statistics over multiple runs. We prefill the model with random input prompts and profile the latency of TTFT. Since input prompt lengths vary in real applications, we do not cache CUDA graphs for the prefilling stage of the model inference.

Citation

@misc{chiang2025elana,
  title={ELANA: A Simple Energy and Latency Analyzer for LLMs},
  author={Chiang et al. (2025)},
  year={2025},
  note={arXiv:2512.09946}
}

• arXiv: 2512.09946