Skillret Eval

skillret-eval

SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents — Cho et al. (2026) (arXiv:2605.05726, 2026)

What this evaluates

Evaluates the ability of embedding models and rerankers to accurately retrieve relevant software skills from a large, noisy library based on long, scenario-rich user queries. It probes ranking quality, recall, and completeness in a two-stage retrieve-then-rerank pipeline, highlighting the need for domain-specific fine-tuning over general semantic matching.

Datasets

• SkillRet — total 17810; splits: train (127190), test (2319); repo https://github.com/ThakiCloud/SKILLRET

Metrics

• NDCG@k (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank k. Measures the quality of the ranked list of retrieved skills, with higher scores indicating better alignment of ground-truth skills at the top of the list.
• Recall@k — range: [0, 1]
  • The fraction of ground-truth skills for a query that appear within the top-k retrieved results.
• Completeness@k — range: [0, 1]
  • The fraction of queries where all ground-truth skills are successfully retrieved within the top-k results (i.e., Recall@k = 1).

Input / output format

Input: User query (typically long, scenario-rich text) and a candidate pool of skill documents (each containing name, description, and full Markdown body).

Output: A ranked list of top-k skill candidates returned by the retriever and/or reranker.

Scoring recipe

def compute_metrics(predictions, gold, k_values=[5, 10, 15]):
    metrics = {}
    for k in k_values:
        top_k = predictions[:k]
        recall = len(set(top_k) & set(gold)) / len(gold)
        completeness = 1.0 if recall == 1.0 else 0.0
        dcg = sum(1.0 / math.log2(i + 2) for i, s in enumerate(top_k) if s in gold)
        idcg = sum(1.0 / math.log2(i + 2) for i in range(len(gold)))
        ndcg = dcg / idcg if idcg > 0 else 0.0
        metrics[f'Recall@{k}'] = recall
        metrics[f'Completeness@{k}'] = completeness
        metrics[f'NDCG@{k}'] = ndcg
    return metrics

Common pitfalls

• Using general-purpose rerankers can degrade performance due to domain mismatch, overriding correct results from specialized retrievers.
• High scores on general benchmarks like MTEB do not predict performance on skill retrieval, as the task requires identifying sparse capability signals in long, noisy queries.
• Models may distribute attention diffusely across entire queries instead of focusing on the few sentences that directly signal skill intent.

Evidence (verbatim from paper)

We report three metrics at $k\in{5,10,15}$: NDCG@$k$ measures ranking quality, Recall@$k$ measures the fraction of ground-truth skills retrieved, and Completeness@$k$ measures the fraction of queries where all ground-truth skills are retrieved, i.e., Recall@$k=1$.

Citation

@misc{cho2026skillret,
  title={SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents},
  author={Cho et al. (2026)},
  year={2026},
  note={arXiv:2605.05726}
}

• arXiv: 2605.05726