Insure Dial Eval

insure-dial-eval

INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection — Kulkarni et al. (2026) (arXiv:2602.18448, 2026)

What this evaluates

Probes phase-aware compliance verification and phase boundary detection in insurance benefit verification calls. It measures a model’s ability to accurately segment conversational phases under workflow-specific rules and apply rule-based compliance reasoning (Information and Procedural Compliance) to fixed spans.

Datasets

• INSURE-Dial — total 1050; splits: real (50), synthetic (1000)

Metrics

• exact match (EM) (primary) — range: percent
  • Fraction of calls where all phases satisfy the task acceptance rules (Task 1) or conjunction over applicable phases (Task 2).
• turn-level F1 — range: [0, 1]
  • Overlap metric measuring turn-level span alignment between predicted and reference boundaries.
• SAD — range: other
  • Sum of absolute differences between predicted and reference span boundaries; lower indicates less boundary drift.
• phase-level accuracy (Acc) — range: percent
  • Fraction of phases correctly classified as compliant or non-compliant given fixed spans.
• macro-F1 — range: [0, 1]
  • Macro-averaged F1 score computed over non-"NA" phases.
• Hit — range: percent
  • Mean fraction of applicable phases correctly identified per call.

Input / output format

Input: Conversational transcript (real or synthetic) annotated with phase-structured JSON schemas capturing information and procedural compliance across IVR, patient identification, coverage status, drug checks, and agent identification.

Output: Predicted span boundaries for each phase (Task 1) and compliance labels (IC/PC) for fixed spans (Task 2), structured according to the phase-annotated JSON schema.

Scoring recipe

# Task 1: Phase Boundary Detection
call_em = 1.0
sad = 0
for pred_span, gold_span in zip(pred_spans, gold_spans):
    if pred_span != gold_span: call_em = 0.0
    sad += abs(len(pred_span) - len(gold_span))
f1 = compute_turn_level_overlap(pred_spans, gold_spans)

# Task 2: Compliance Verification (given fixed spans)
applicable = [i for i, m in enumerate(applicability_mask) if m]
correct = sum(pred_labels[i] == gold_labels[i] for i in applicable)
acc = correct / len(applicable)
hit = correct / len(applicable)
call_acc = 1.0 if correct == len(applicable) else 0.0
macro_f1 = compute_macro_f1(pred_labels, gold_labels, exclude_na=True)

Common pitfalls

• Call-level EM collapses due to multiplicative strictness across 8–10 phases; a single boundary slip fails the entire call.
• IC and PC call-level accuracies are computed over different overall fields with different applicability patterns, making them incomparable across tasks.
• Synthetic data omits long hold-time chatter and IVR repetitions, inflating segmentation scores compared to real calls.

Evidence (verbatim from paper)

For Task 1 we report exact match (EM) of the predicted span under §4.1 rules, turn-level overlap via $F_{1}$ , and sum of absolute differences (SAD) as a boundary-drift measure (lower is better). Call-level EM is the fraction of calls where all phases (including correctly predicting null when absent) satisfy the Task 1 acceptance rules. For Task 2 we report phase-level accuracy (Acc) and macro- $F_{1}$ over non-"NA" phases, plus Hit, the mean fraction of applicable phases correct per call

Citation

@misc{kulkarni2026insuredial,
  title={INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection},
  author={Kulkarni et al. (2026)},
  year={2026},
  note={arXiv:2602.18448}
}

• arXiv: 2602.18448