qhjqhj00 terminal-bench-2.0-eval
On Data Engineering for Scaling LLM Terminal Capabilities — Pi et al. (2026) (arXiv:2602.21193, 2026)
What this evaluates
Evaluates an LLM's ability to execute complex, multi-step terminal commands and tasks in a sandboxed environment. It probes capabilities across software engineering, system administration, data processing, security, and debugging.
Datasets
- Terminal-Bench 2.0 — total ?; splits: test (-1)
Metrics
TB2.0(primary) — range: percent- Mean success rate across tasks, reported as average ± standard deviation over multiple runs. Tasks are evaluated in isolated cloud sandboxes using the Daytona framework, which checks for successful completion of terminal commands and task objectives.
Input / output format
Input: Natural language task descriptions requiring multi-step terminal commands and file manipulations.
Output: Sequences of terminal commands and actions executed within a containerized environment.
Scoring recipe
results = []
for task in test_set:
sandbox = Daytona.create_sandbox()
trajectory = model.generate(task.prompt)
success = sandbox.execute_and_verify(trajectory, task.checkpoints)
results.append(success)
mean_score = sum(results) / len(results)
std_score = std(results)
return mean_score, std_score
Common pitfalls
- Strict filtering of training data (e.g., success-only) discards valuable error-recovery patterns and significantly hurts performance.
- Extending context length beyond 32k tokens or using YaRN2 scaling does not improve TB2.0 scores and may slightly hurt performance due to noisy long-tail trajectories.
Evidence (verbatim from paper)
We evaluate Terminal-Task-Gen by benchmarking Nemotron-Terminal on Terminal-Bench 2.0 (TB2.0). As shown in Table [3], our models demonstrate substantial gains: Nemotron-Terminal-8B achieves $13.0\pm 2.2$, a five-fold increase over Qwen3-8B ($2.47\pm 0.5$).
Citation
@misc{pi2026dataengineering,
title={On Data Engineering for Scaling LLM Terminal Capabilities},
author={Pi et al. (2026)},
year={2026},
note={arXiv:2602.21193}
}
- arXiv: 2602.21193