# Terminal Bench Agent Evaluation

> Evaluate agents on 89 challenging terminal-based tasks where frontier models score below 65%, providing realistic assessment of command-line interface automation capability. Use when benchmarking agents designed for system administration, automation, or DevOps tasks.

- Skill: `adu2021/terminal-bench-agent-evaluation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add adu2021/terminal-bench-agent-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/terminal-bench-agent-evaluation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: adu2021 (https://skillmd.com/u/adu2021)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/adu2021/terminal-bench-agent-evaluation

---


# Terminal-Bench: Agent Evaluation on CLI Tasks

This skill provides a comprehensive benchmarking suite for evaluating agents on realistic, hard terminal-based tasks, revealing significant gaps in even frontier models' abilities to automate command-line workflows.

## When to Use
- Evaluating CLI automation agents (DevOps, system administration)
- Measuring agent capability on realistic computer-use tasks
- Identifying gaps in agent reasoning for terminal-based workflows
- Benchmarking improvements in agent autonomy and problem-solving
- Comparing agent performance across different models/architectures

## When NOT to Use
- Tasks not involving command-line interfaces
- Simple command execution (Terminal-Bench is for complex workflows)
- Systems where safety is critical (tests include risky operations)
- Evaluating GUI-based agents (Terminal-Bench is CLI-specific)

## Key Concept
Terminal-Bench 2.0 contains 89 challenging tasks across realistic scenarios:
- File system operations (finding, organizing, searching)
- Process management (monitoring, controlling, debugging)
- Network operations (connectivity, configuration)
- Software installation and management
- Log analysis and troubleshooting

These tasks reflect real-world challenges where even frontier models fail < 65% of the time.

## Benchmark Task Categories

Terminal-Bench covers:
- **File Management**: Complex operations requiring understanding of file types, permissions, search patterns
- **System Monitoring**: Extracting metrics from multiple sources, identifying patterns
- **Configuration**: Managing system settings, understanding configuration formats
- **Troubleshooting**: Diagnosing issues from partial information, using multiple diagnostic tools
- **Scripting**: Creating bash scripts for complex multi-step workflows

## Using the Benchmark

Evaluate an agent on Terminal-Bench tasks:

```python
# Pseudocode for Terminal-Bench evaluation
class TerminalBench:
    def __init__(self, tasks_path="terminal_bench_tasks.json"):
        self.tasks = load_tasks(tasks_path)  # 89 tasks

    def evaluate_agent(self, agent, timeout_per_task=300):
        results = {"passed": 0, "failed": 0, "details": []}

        for task in self.tasks:
            try:
                # Execute task with timeout
                output = agent.execute_terminal_task(task.description)

                # Validate output against expected results
                if self.validate_output(output, task.expected_output):
                    results["passed"] += 1
                    results["details"].append({
                        "task": task.id,
                        "success": True
                    })
                else:
                    results["failed"] += 1
                    results["details"].append({
                        "task": task.id,
                        "success": False,
                        "reason": "output validation failed"
                    })
            except TimeoutError:
                results["failed"] += 1

        return results
```

## Key Insights
- Frontier models achieve < 65% success rate on Terminal-Bench tasks
- Clear gap between simple command execution and complex workflows
- Multi-step reasoning and tool composition remain challenging
- Real-world CLI automation is significantly harder than assumed

## Research Context
Terminal-Bench 2.0 is designed to reveal genuine limitations of agent reasoning on realistic computer-use tasks. The high difficulty (most frontier models fail 35%+ of tasks) demonstrates that CLI automation remains an open challenge despite advances in other agent domains.