Gap Overlap Kg Eval

gap-overlap-kg-eval

A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness — Mridul et al. (2026) (arXiv:2604.10853, 2026)

What this evaluates

Evaluates a knowledge graph's ability to perform gap and overlap analysis on life insurance contracts by answering scenario-based competency questions. It probes the system's capacity for structured, evidence-grounded reasoning to determine claim coverage, denial, or non-applicability across heterogeneous contract types.

Datasets

• Insurance Contract KG Benchmark — total 58; splits: test (58); repo https://github.com/brains-group/gap_and_overlap_analysis_insurance_contract_benchmark

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact match accuracy between the model's predicted claim status (COVERED, DENIED, or NOT_APPLICABLE) and the ground truth for each contract-scenario pair, averaged across all 58 scenarios and 10 contracts.

Input / output format

Input: Natural language scenario description (e.g., 'Insured dies by suicide exactly 13 months after the policy issue date. All premiums paid on time.') provided as a JSON prompt, along with access to the ontology (TBox/ABox) and contract corpus.

Output: JSON object containing the predicted claim determination (COVERED, DENIED, or NOT_APPLICABLE) for each of the 10 contracts, optionally accompanied by the generated SPARQL query and traceability evidence (sourceContractID, sourceContractSection, coverageSourceText).

Scoring recipe

correct = 0
total = 0
for scenario in test_set:
    pred = model.generate(scenario.prompt)
    for contract_id in contracts:
        if pred[contract_id].status == scenario.ground_truth[contract_id].status:
            correct += 1
        total += 1
return correct / total

Common pitfalls

• Relying on surface-level string matching instead of ontology structure or typed properties leads to false negatives for parametric differences (e.g., varying exclusion periods or benefit types).
• Failing to provide traceability evidence (sourceContractID, sourceContractSection) makes results non-auditable and violates the benchmark's explainability requirement.
• Treating NOT_APPLICABLE as equivalent to DENIED, which skews accuracy on edge cases like simultaneous death or lapse during grace periods.

Evidence (verbatim from paper)

By providing two SPARQL queries per scenario, we enable a robust deterministic evaluation that measures the accuracy of both positive and negative claim determinations with evidence-based traceability.

Citation

@misc{mridul2026gapoverlap,
  title={A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness},
  author={Mridul et al. (2026)},
  year={2026},
  note={arXiv:2604.10853}
}

• arXiv: 2604.10853