Backbench Eval

backbench-eval

Human-Guided Harm Recovery for Computer Use Agents — Li et al. (2026) (arXiv:2604.18847, 2026)

What this evaluates

This benchmark probes an agent's ability to recover from harmful states in real-world computer use environments by backtracking or remediating to a safe operational state. It evaluates how well agents align with human preferences during recovery under varying resource constraints (step limits).

Datasets

• BackBench — total 50; splits: test (50)

Metrics

• Bradley-Terry rating (primary) — range: other
  • Aggregated from human A/B pairwise preferences using a Bradley-Terry model. The strength parameter p_i estimates the probability that system i beats system j. Converted to an interpretable scale via R = 1500 + 400 * log10(p_i), analogous to chess ratings.

Input / output format

Input: Initial prompt specifying the task and step limit, plus the corresponding initial system state (a harmful scenario instantiated in an Ubuntu-based GUI environment).

Output: Complete agent trajectory (sequence of actions taken to recover from the harm and return to a safe state).

Scoring recipe

def compute_bradley_terry_ratings(system_trajectories, human_judgments):
    # human_judgments contains pairs of trajectories and annotator choices
    pairwise_wins = count_wins(system_i, system_j, human_judgments)
    # Fit Bradley-Terry model via maximum likelihood estimation
    p_i = max_likelihood_estimate(pairwise_wins)
    # Convert to interpretable rating scale
    rating_i = 1500 + 400 * math.log10(p_i)
    # Bootstrap resampling (n=1000) to estimate standard errors
    return rating_i, bootstrap_se(rating_i)

Common pitfalls

• Step limits (15 vs 50 steps) significantly alter optimal recovery trajectories; evaluating them together may mask performance differences under resource constraints.
• The metric is relative and pairwise; absolute ratings depend on the specific set of competing systems included in the comparison pool.
• Human annotators evaluate complete action sequences, not just final states, requiring precise logging of intermediate steps for fair A/B comparison.

Evidence (verbatim from paper)

Accordingly, we adopt a comparative A/B preference framework for evaluation: a human annotator is shown pairs of complete agent trajectories—two alternative sequences of actions taken to recover from the same harm—and asked to decide which trajectory is superior. These pairwise judgments are then aggregated using an Bradley-Terry rating system, yielding relative performance scores across all evaluated scaffolds.

Citation

@misc{li2026humanguidedharmrecovery,
  title={Human-Guided Harm Recovery for Computer Use Agents},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.18847}
}

• arXiv: 2604.18847