Codetracer Eval

codetracer-eval

CodeTracer: Towards Traceable Agent States — Han Li et al. (2026) (arXiv:2604.11641, 2026)

What this evaluates

This benchmark evaluates an agent's ability to localize the onset of failure within long-horizon code execution trajectories by analyzing heterogeneous run artifacts. It probes how well models can distinguish genuinely failure-relevant steps from salient but irrelevant logs, diagnose execution bottlenecks, and recover from early wrong commitments under constrained token budgets.

Datasets

• CodeTraceBench — total 3326; splits: test (3326); repo https://github.com/NJU-LINK/CodeTracer

Metrics

• step-level Precision — range: percent
  • TP / (TP + FP), where TP and FP are correctly and incorrectly identified failure steps at the step level.
• step-level Recall — range: percent
  • TP / (TP + FN), where FN are missed failure steps.
• step-level F1 (primary) — range: percent
  • 2 * (Precision * Recall) / (Precision + Recall), macro-averaged across difficulty, category, and execution stage splits.
• token usage — range: other
  • Total tokens consumed by the model during the localization and replay phases.

Input / output format

Input: Heterogeneous run directories/logs containing execution artifacts from agent frameworks, optionally pre-processed into standardized traces or hierarchical trace trees.

Output: A set of identified step indices or stage labels corresponding to the earliest critical failure point(s) in the trajectory.

Scoring recipe

def compute_step_metrics(predictions, gold):
    tp = len(predictions & gold)
    fp = len(predictions - gold)
    fn = len(gold - predictions)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
# Macro-average across splits (Easy/Hard, categories, stages)

Common pitfalls

• Confusing salient or noisy log artifacts with genuinely failure-relevant steps, leading to high recall but low precision.
• Token budget constraints directly dictate search depth; models may terminate prematurely on hard tasks, missing dispersed failure evidence.
• Step-level granularity requires precise alignment between predicted steps and ground-truth annotations, which can be sensitive to trajectory parsing variations.

Evidence (verbatim from paper)

We report macro averaged step level Precision, Recall, and F1 (formal definitions in [Appendix C]) and total token usage. Across all backbones, adding structure and tracing signals substantially improves localization quality over raw log prompting. The three frontier backbones reach comparable F1 (46–48%) but diverge in how they traverse the trace.

Citation

@misc{hanli2026codetracer,
  title={CodeTracer: Towards Traceable Agent States},
  author={Han Li et al. (2026)},
  year={2026},
  note={arXiv:2604.11641}
}

• arXiv: 2604.11641