Disrouter Routing Eval

disrouter-routing-eval

DiSRouter: Distributed Self-Routing for LLM Selections — Zheng et al. (2025) (arXiv:2510.19208, 2025)

What this evaluates

Evaluates a distributed multi-agent routing system's ability to autonomously assign queries to the most appropriate LLM based on intrinsic self-assessment. It probes the trade-off between routing accuracy and inference cost across diverse mathematical, commonsense, and reading comprehension benchmarks.

Datasets

• GSM8K — total ?; splits: test (-1)
• ARC — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• RACE_HIGH — total ?; splits: test (-1)
• OpenbookQA — total ?; splits: test (-1)
• DROP — total ?; splits: test (-1)
• CosmosQA — total ?; splits: test (-1)
• SQuAD — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• HeadQA — total ?; splits: test (-1)

Metrics

• accuracy — range: [0, 1]
  • Standard exact-match or option-matching accuracy computed per dataset. Averaged across datasets for final reporting.
• cost — range: [0, 1]
  • Average normalized inference cost of the routed models per query. Each model is assigned a fixed normalized cost between 0 and 1 based on its size.
• utility (primary) — range: [0, 1]
  • A tunable weighted metric balancing accuracy and cost, controlled by a global preference factor α. Higher α prioritizes cost reduction, while lower α prioritizes accuracy.

Input / output format

Input: Natural language query or task prompt from the benchmark dataset, evaluated in a zero-shot Chain-of-Thought manner.

Output: Routing decision specifying which LLM from the pool handles the query, followed by the LLM's generated response or a rejection signal.

Scoring recipe

def compute_metrics(predictions, routing_decisions, golds, model_costs, alpha):
    acc = sum(1 for p, g in zip(predictions, golds) if is_correct(p, g)) / len(golds)
    cost = sum(model_costs[m] for m in routing_decisions) / len(golds)
    utility = alpha * acc + (1 - alpha) * (1 - cost)
    return acc, cost, utility

Common pitfalls

• Uses validation sets as test sets for benchmarks with hidden test splits (e.g., GSM8K, MMLU), which may inflate or deflate reported accuracy compared to official leaderboards.
• Cost is a normalized fixed value per model rather than actual latency or token usage, making cross-system cost comparisons sensitive to the normalization scheme.
• The utility metric is highly sensitive to the global preference factor α; results reported for one α value do not generalize to other scenarios without re-evaluation.

Evidence (verbatim from paper)

We evaluate LLMs in a zero-shot Chain-of-Thought (CoT, Wei et al., [2022]) manner and use accuracy as the evaluation metric for all these datasets. To evaluate the overall routing performance, we utilize the utility metric introduced in §[2.1], Equation ([2]), which can be adjusted for different scenarios by setting a corresponding value for α.

Citation

@misc{zheng2025disrouter,
  title={DiSRouter: Distributed Self-Routing for LLM Selections},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2510.19208}
}

• arXiv: 2510.19208