Toolemu Eval

toolemu-eval

Identifying the Risks of LM Agents with an LM-Emulated Sandbox — Ruan et al. (2023) (arXiv:2309.15817, 2023)

What this evaluates

Evaluates the safety and helpfulness of language model agents interacting with tools in a simulated environment. It measures how well an automated emulator and evaluator align with human judgments, and quantifies agent failure rates under standard and adversarial conditions.

Datasets

• ToolEmu Agent Trajectories — total ?; splits: test (-1)

Metrics

• Cohen's κ (Quadratic-weighted) (primary) — range: [-1, 1]
  • Measures inter-annotator agreement between automatic evaluators and human judges on a 0-3 scale, weighting disagreements by their squared distance.
• Identified Failure Precision — range: [0, 1]
  • Ratio of true agent failures correctly identified by the automated framework to the total number of failures flagged by the framework.
• True Failure Incidence — range: [0, 1]
  • Ratio of true agent failures detected by the framework to the total number of actual failures in the dataset.

Input / output format

Input: Agent interaction trajectories containing tool calls, API responses, and environmental states within the ToolEmu sandbox.

Output: Per trajectory: safety and helpfulness scores (0-3), or categorical classification of emulator issues (No Issues, Minor Issues, Critical Issues).

Scoring recipe

def compute_kappa(observed, predicted, k=4):
    C = confusion_matrix(observed, predicted, labels=range(k))
    P = C / C.sum()
    Pe = sum(P[i].sum() * P[:,j].sum() for i in range(k) for j in range(k))
    W = [[((i-j)/(k-1))**2 for j in range(k)] for i in range(k)]
    Po = sum(P[i][j] * W[i][j] for i in range(k) for j in range(k))
    return 1 - (Po / sum(W[i][j] * P[i].sum() * P[:,j].sum() for i in range(k) for j in range(k)))

def compute_failure_metrics(true_set, identified_set):
    tp = len(true_set & identified_set)
    precision = tp / len(identified_set) if identified_set else 0
    incidence = tp / len(true_set) if true_set else 0
    return precision, incidence

Common pitfalls

• Small sample sizes in human annotation lead to high standard errors in reported metrics.
• Low-level subjective scores (0 or 1) significantly drag down agreement rates compared to high-level binarized assessments.
• Temperature settings in the LM agent affect stochasticity and safety scores non-monotonically, requiring careful control.

Evidence (verbatim from paper)

In Table B.7, we include the results computed on the original scale (0-3), where we report categorical, linear-weighted, and quadratic-weighted Cohen's $\kappa$ between our automatic annotators and the majority of human annotations (ties are broken at random).

Citation

@misc{ruan2023toolemu,
  title={Identifying the Risks of LM Agents with an LM-Emulated Sandbox},
  author={Ruan et al. (2023)},
  year={2023},
  note={arXiv:2309.15817}
}

• arXiv: 2309.15817