Skillflow Eval

skillflow-eval

SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents — Zhang et al. (2026) (arXiv:2604.17308, 2026)

What this evaluates

Evaluates autonomous agents' ability to discover, patch, and evolve reusable skills over time in a sequential, lifelong learning setting. It probes whether models can consolidate successful execution traces into a compact, repairable skill library rather than merely accumulating fragmented task-specific traces.

Datasets

• SkillFlow — total 166; splits: test (166); repo https://github.com/ZhangZi-a/SkillFlow

Metrics

• task completion rate (%comp.) (primary) — range: percent
  • Percentage of tasks successfully completed within a task family. Calculated as (number of completed tasks / total tasks) * 100.
• interaction turns (Turns) — range: other
  • Average number of interaction turns per task.
• cost (Cost) — range: other
  • Average USD cost per task.
• output tokens (Out Tok.) — range: other
  • Average output tokens (in thousands) per task.
• skill usage rate (%use) — range: percent
  • Percentage of tasks that read or call at least one stored skill from the library.

Input / output format

Input: Task instructions/prompts from the benchmark, optionally preceded by a skill library of previously generated artifacts.

Output: Task solutions and, upon completion or failure, a single-turn skill-patch artifact (code or instructions) to be added to the library.

Scoring recipe

def compute_metrics(predictions, gold, library, turns, costs, tokens):
    completed = sum(1 for p, g in zip(predictions, gold) if p == g)
    comp_rate = (completed / len(gold)) * 100
    avg_turns = sum(turns) / len(turns)
    avg_cost = sum(costs) / len(costs)
    avg_tokens = sum(tokens) / len(tokens)
    skills_used = sum(1 for t in tasks if t.reads_or_calls_skill(library))
    use_rate = (skills_used / len(tasks)) * 100
    return {
        '%comp.': comp_rate,
        'Turns': avg_turns,
        'Cost': avg_cost,
        'Out Tok.(K)': avg_tokens,
        '#Skills': len(library),
        '%use': use_rate
    }

Common pitfalls

• High skill usage or large library size does not guarantee better performance; fragmented skill inflation often degrades efficiency.
• Incorrect early skills can cause systematic downstream drift, making later tasks harder rather than easier.
• Gains are highly model-dependent; some models show negative transfer despite skill evolution.

Evidence (verbatim from paper)

We evaluate SkillFlow under the Agentic Lifelong Learning protocol. Within each task family, an agent begins with an empty skill library, solves the first task without prior skills, and then updates the library iteratively through skill patches derived from execution trajectories and rubric feedback. %comp. is task completion rate; Turns, Cost, and Out Tok. are per-task averages of interaction turns, USD cost, and output tokens (thousands). #Skills is the cumulative number of skills generated within a task family, and %use is the percentage of tasks that read or call at least one stored skill.

Citation

@misc{zhang2026skillflow,
  title={SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.17308}
}

• arXiv: 2604.17308