Skilllearnbench Eval

skilllearnbench-eval

SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks — Zhong et al. (2026) (arXiv:2604.20087, 2026)

What this evaluates

This benchmark evaluates continual learning methods for generating reusable procedural skills in LLM agents. It probes the quality of generated skills, their alignment with execution trajectories, and the ultimate task-solving accuracy and efficiency of a fixed solving agent.

Datasets

• SkillLearnBench — total 20; splits: test (20); repo https://github.com/cxcscmu/SkillLearnBench

Metrics

• Acc. (primary) — range: percent
  • Task accuracy, calculated as the percentage of tasks solved correctly by the solving agent out of the total tasks evaluated.
• #Tokens — range: other
  • Total number of tokens consumed by the solving agent during task execution across all turns.
• Coverage — range: percent
  • Percentage of core skill components correctly identified or generated, evaluated by an LLM judge.
• Executability — range: percent
  • Percentage of generated skills that can be successfully executed in the sandbox environment.
• Safety — range: percent
  • Percentage of generated skills that meet predefined safety criteria, evaluated by an LLM judge.
• Alignment — range: percent
  • Trajectory alignment score evaluating how well the agent's execution behavior matches the skill's intent.
• Usage — range: percent
  • Skill usage rate measuring how frequently the solving agent adopts the generated skill during execution.

Input / output format

Input: Task descriptions from SkillLearnBench, provided to a fixed solving agent (Claude Sonnet 4.6) operating in a containerized sandbox with task-specific tools.

Output: Generated procedural skills (for continual learning methods) or task execution trajectories/solutions (for the solving agent).

Scoring recipe

def compute_metrics(execution_logs, ground_truth):
    correct = 0
    total_tokens = 0
    for log, gt in zip(execution_logs, ground_truth):
        total_tokens += log.token_count
        if log.is_correct(gt):
            correct += 1
    accuracy = (correct / len(ground_truth)) * 100
    return {'Acc.': accuracy, '#Tokens': total_tokens}

Common pitfalls

• Assuming stronger generation LLMs consistently yield better skills; the paper shows method rankings shift across LLM families and stronger models do not reliably improve skill quality.
• Focusing solely on skill content while ignoring skill adoption; the paper notes that task success depends heavily on whether the solving agent actually adopts the generated skill (Usage rate).
• Confusing the continual learning LLM (which generates skills) with the fixed solving LLM (Claude Sonnet 4.6); evaluation isolates skill quality by keeping the solver constant.

Evidence (verbatim from paper)

Trajectory alignment score (Alignment) and skill usage rate (Usage) evaluate execution behavior. Solving token cost (#Tokens) and accuracy (Acc.) measure task outcome.

Citation

@misc{zhong2026skilllearnbench,
  title={SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks},
  author={Zhong et al. (2026)},
  year={2026},
  note={arXiv:2604.20087}
}

• arXiv: 2604.20087