First Inference Eval

first-inference-eval

FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access — Tanikanti et al. (2025) (arXiv:2510.13724, 2025)

What this evaluates

Evaluates the performance and scalability of a federated inference scheduling framework under varying request loads, measuring throughput, latency, and auto-scaling capabilities on distributed HPC resources.

Datasets

• ShareGPT — total 1000; splits: test (1000)

Metrics

• Request throughput (req/s) (primary) — range: other
  • The number of successful inference requests completed per second. Measures the overall request processing capacity of the system.
• Output Token Throughput (tok/s) — range: other
  • The total number of output tokens generated by the model per second across all successful requests. Reflects the effective generation speed.
• Median End-to-End Latency (s) — range: other
  • The time elapsed from sending a request to receiving the complete response. Reported as the median to reduce sensitivity to outliers compared to the mean.
• Benchmark Duration (s) — range: other
  • The total time taken to complete all requests in a given benchmark run. Provides an overall measure of workload processing speed.

Input / output format

Input: Prompts from the ShareGPT dataset, sent as OpenAI-compatible API requests with specified target output lengths.

Output: Generated text responses (tokens) from the LLM, returned via the API.

Scoring recipe

def compute_metrics(predictions, gold):
    latencies = [r.timestamp_received - r.timestamp_sent for r in predictions]
    median_latency = sorted(latencies)[len(latencies)//2]
    total_time = max(r.timestamp_received for r in predictions) - min(r.timestamp_sent for r in gold)
    req_throughput = len(predictions) / total_time
    tok_throughput = sum(r.output_tokens for r in predictions) / total_time
    return {
        'request_throughput': req_throughput,
        'output_token_throughput': tok_throughput,
        'median_latency': median_latency,
        'benchmark_duration': total_time
    }

Common pitfalls

• Confusing median latency with mean latency; the protocol explicitly reports median to avoid outlier sensitivity.
• Ignoring the 'infinite request rate' workload condition; peak performance tests saturate the server by sending all 1000 requests immediately, differing from steady-state load testing.
• Comparing FIRST directly to commercial APIs without accounting for external rate limits and hardware differences (HPC vs. cloud).

Evidence (verbatim from paper)

We focus on the following key metrics (Sagi, [2025]) to evaluate performance: Request throughput (req/s): The number of successful inference requests completed per second. This metric measures the overall request processing capacity of the system. Output Token Throughput (tok/s): The total number of output tokens generated by the model per second across all successful requests. This metric reflects the effective generation speed. Median End-to-End Latency (s): The time elapsed from sending a request to receiving the complete response. We primarily report the median latency, as it is less sensitive to outliers than the mean. This metric reflects the user-perceived responsiveness for a single request.

Citation

@misc{tanikanti2025first,
  title={FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access},
  author={Tanikanti et al. (2025)},
  year={2025},
  note={arXiv:2510.13724}
}

• arXiv: 2510.13724