Skillrouter Eval

skillrouter-eval

SkillRouter: Retrieve-and-Rerank Skill Selection for LLM Agents at Scale — YanZhao Zheng et al. (2026) (arXiv:2603.22455, 2026)

What this evaluates

This benchmark evaluates the ability of LLM-based skill routers to accurately select the most relevant tool or function from a large-scale pool of candidate skills based on a user query. It probes retrieval and reranking capabilities under varying difficulty tiers and tests whether models can handle single-skill versus multi-skill routing scenarios.

Datasets

• SkillRouter Benchmark — total 75; splits: test (75)

Metrics

• Hit@1 (primary) — range: [0, 1]
  • Primary top-1 routing metric. Counts as success if any required skill is ranked first. For multi-skill queries, success is recorded when at least one ground-truth skill appears at rank 1. [0, 1]
• MRR@10 — range: [0, 1]
  • Mean Reciprocal Rank at 10. Computes the average of 1/rank for the first relevant skill in the top-10 results. [0, 1]
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at 10. Measures ranking quality by discounting the relevance of correctly retrieved skills by their position, normalized by the ideal DCG. [0, 1]
• Recall@10 — range: [0, 1]
  • Fraction of ground-truth skills retrieved within the top-10 candidates. Computed as |predicted[:10] ∩ gold| / |gold|. [0, 1]
• Recall@20 — range: [0, 1]
  • Fraction of ground-truth skills retrieved within the top-20 candidates. Computed as |predicted[:20] ∩ gold| / |gold|. [0, 1]
• Recall@50 — range: [0, 1]
  • Fraction of ground-truth skills retrieved within the top-50 candidates. Computed as |predicted[:50] ∩ gold| / |gold|. [0, 1]
• FC@10 — range: [0, 1]
  • Full Coverage at 10. Indicates whether all required skills for a multi-skill query are retrieved within the top-10 candidates. [0, 1]

Input / output format

Input: A user query paired with a list of candidate skills. Each skill is represented by three text fields: name, description, and body. For retrievers, the full skill pool (~80K) is used; for rerankers, the top-20 candidates from the retriever are provided.

Output: A ranked list of candidate skills (or a single top-1 skill selection for LLM-as-judge baselines).

Scoring recipe

def compute_metrics(predictions, gold_skills):
    # predictions: list of skill IDs ranked by model
    # gold_skills: set of required skill IDs for the query
    hit1 = 1.0 if any(s in gold_skills for s in predictions[:1]) else 0.0
    rr = [1.0/(i+1) for i, s in enumerate(predictions) if s in gold_skills]
    mrr = rr[0] if rr else 0.0
    recall10 = len(set(predictions[:10]) & gold_skills) / len(gold_skills)
    recall20 = len(set(predictions[:20]) & gold_skills) / len(gold_skills)
    recall50 = len(set(predictions[:50]) & gold_skills) / len(gold_skills)
    fc10 = 1.0 if gold_skills.issubset(set(predictions[:10])) else 0.0
    return hit1, mrr, recall10, recall20, recall50, fc10

Common pitfalls

• Ignoring the single-skill vs. multi-skill distinction, which causes Hit@1 and coverage metrics (FC@10) to diverge significantly.
• Using only name and description (nd) inputs for rerankers actively degrades performance below encoder-only retrieval due to insufficient context.
• Assuming zero-shot general-purpose encoders outperform domain-adapted fine-tuned models without empirical verification on skill routing data.

Evidence (verbatim from paper)

We report seven metrics: Hit@1 (our primary top-1 routing metric), MRR@10 (mean reciprocal rank), nDCG@10 (normalized discounted cumulative gain), Recall@10, Recall@20, Recall@50, and FC@10. For multi-skill queries, Hit@1 counts success when any required skill is ranked first, while Recall@10 and FC@10 provide complementary coverage views.

Citation

@misc{zheng2026skillrouter,
  title={SkillRouter: Retrieve-and-Rerank Skill Selection for LLM Agents at Scale},
  author={YanZhao Zheng et al. (2026)},
  year={2026},
  note={arXiv:2603.22455}
}

• arXiv: 2603.22455