Policy Selection Eval

policy-selection-eval

DoubleAgents: Exploring Mechanisms of Building Trust with Proactive AI — Long et al. (2025) (arXiv:2509.12626, 2025)

What this evaluates

Evaluates the model's ability to retrieve relevant coordination policies that guide task planning based on a high-level progress summary of the current state.

Datasets

• Policy Selection Test Suite — total 30; splits: test (30)

Metrics

• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall computed over the set of retrieved policies versus the ground truth set of user-approved policies.

Input / output format

Input: A high-level progress summary of the task state (e.g., speaker responsiveness, slot availability, and outreach history) generated by the getProgressSummary() API.

Output: A set of applicable coordination policies retrieved from a curated seed set.

Scoring recipe

def compute_f1(predictions, gold):
    pred_set = set(predictions)
    gold_set = set(gold)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Raw aggregated context (speaker + slot) without summarization yields very low F1 (0.27) compared to summary-based prompting (0.70).
• Evaluations use temperature=1.0 with o4-mini, which may increase variability; results are averaged over 3 runs.

Evidence (verbatim from paper)

we constructed a test suite of 30 examples, all sourced from real user study sessions and curated to reflect a diverse set of scheduling scenarios. For each example, we treat the set of policies approved by users during the study as the ground truth for evaluation. All evaluations were conducted using the o4-mini model with a default temperature of 1. We report the average F1 score across three independent runs for each experimental setting.

Citation

@misc{long2025doubleagents,
  title={DoubleAgents: Exploring Mechanisms of Building Trust with Proactive AI},
  author={Long et al. (2025)},
  year={2025},
  note={arXiv:2509.12626}
}

• arXiv: 2509.12626