Edge Case Detection Eval

edge-case-detection-eval

DoubleAgents: Exploring Mechanisms of Building Trust with Proactive AI — Long et al. (2025) (arXiv:2509.12626, 2025)

What this evaluates

Tests the model's ability to classify whether a respondent's message represents an edge case that falls outside the scope of existing coordination policies and requires user escalation.

Datasets

• Edge Case Detection Test Suite — total 100; splits: test (100)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified edge cases (binary classification) out of the total test examples.

Input / output format

Input: A respondent's message, optionally accompanied by few-shot demonstration examples and/or a reference list of existing coordination policies.

Output: Binary classification label indicating whether the message is an edge case requiring escalation.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Zero-shot prompting without demonstrations performs significantly worse (0.79) than few-shot approaches (0.88-0.93).
• Experiments are run with GPT-4o at temperature=0.7 and max_tokens=100; changing these may alter performance.

Evidence (verbatim from paper)

we constructed a test suite of 100 examples, manually validated by the lead authors to ensure coverage and quality. The construction methodology for this test suite is detailed in Appendix[B.2]. All experiments were conducted using GPT-4o, with temperature set to 0.7 and a maximum token limit of 100. We report the average accuracy across three independent runs for each condition.

Citation

@misc{long2025doubleagents,
  title={DoubleAgents: Exploring Mechanisms of Building Trust with Proactive AI},
  author={Long et al. (2025)},
  year={2025},
  note={arXiv:2509.12626}
}

• arXiv: 2509.12626