Pick And Spin Eval

pick-and-spin-eval

Efficient Multi-Model Orchestration for Self-Hosted Large Language Models — Vangala et al. (2025) (arXiv:2512.22402, 2025)

What this evaluates

Evaluates a Kubernetes-based multi-model orchestration framework for self-hosted LLMs, measuring how hybrid routing and adaptive scaling affect inference reliability, latency, GPU utilization, and cost across diverse benchmarks.

Datasets

• HumanEval, GSM8K, MBPP, TruthfulQA, ARC, HellaSwag, MATH, MMLU Pro — total 31000; splits: test (31000)

Metrics

• success (primary) — range: percent
  • Proportion of inference runs that returned valid completions within predefined time and token limits. Explicitly measures inference reliability rather than task correctness.
• routing_accuracy — range: percent
  • Percentage of queries correctly routed to the appropriate model tier (L1–L3) based on complexity estimation using keyword heuristics or DistilBERT classification.
• latency — range: seconds | percent
  • Mean inference latency in seconds, or percentage reduction relative to the baseline profile. Includes orchestration and scaling overhead.
• cost_per_query — range: USD
  • Average inference cost in USD per query, calculated across different deployment profiles (static vs. dynamic/KEDA scaling).
• routing_efficiency_eta — range: other
  • Defined as η = (A_r / A_b) / (C_r / C_b), where A_r and A_b are routed and baseline accuracies, and C_r and C_b are their corresponding inference costs. Measures accuracy gain per unit cost overhead.

Input / output format

Input: Text prompts from eight standard benchmarks (HumanEval, GSM8K, MBPP, TruthfulQA, ARC, HellaSwag, MATH, MMLU Pro).

Output: Model-generated text completions, evaluated for validity within predefined time and token limits.

Scoring recipe

def compute_metrics(predictions, gold, config):
    # Success: valid completion within time/token limits
    success = sum(1 for p in predictions if is_valid(p, config.time_limit, config.token_limit))
    success_rate = (success / len(predictions)) * 100
    
    # Routing accuracy: correct model tier assignment
    routing_acc = sum(1 for p, g in zip(predictions, gold) if p.tier == g.expected_tier) / len(predictions) * 100
    
    # Latency & Cost: measured directly from system logs
    avg_latency = sum(p.latency for p in predictions) / len(predictions)
    avg_cost = sum(p.cost for p in predictions) / len(predictions)
    
    # Routing efficiency
    eta = (routing_acc / baseline_acc) / (avg_cost / baseline_cost)
    return success_rate, routing_acc, avg_latency, avg_cost, eta

Common pitfalls

• The paper explicitly defines 'success' as inference reliability (valid completion within time/token limits), not task correctness/accuracy, which differs from standard benchmark reporting.
• Routing accuracy percentages are reported as very low (4.8%–8.6%), likely reflecting strict tier-matching criteria rather than standard classification accuracy, which may confuse readers expecting higher values.
• Latency and cost metrics are system-level measurements (including orchestration overhead and KEDA scaling delays) rather than pure model inference times.

Evidence (verbatim from paper)

Success indicates valid completion within time and token limits, measuring inference reliability rather than task correctness.

Citation

@misc{vangala2025efficient,
  title={Efficient Multi-Model Orchestration for Self-Hosted Large Language Models},
  author={Vangala et al. (2025)},
  year={2025},
  note={arXiv:2512.22402}
}

• arXiv: 2512.22402