Tabular QA Confidence Eval

tabular-qa-confidence-eval

Calibrated Confidence Estimation for Tabular Question Answering — Voss (2026) (arXiv:2604.12491, 2026)

What this evaluates

Evaluates the calibration and reliability of confidence scores produced by LLMs when answering questions over tabular data. It probes how well predicted confidence aligns with actual accuracy across different elicitation methods and dataset complexities.

Datasets

• WikiTableQuestions — total 2000; splits: val (2000)
• TableBench — total 836; splits: test (836)

Metrics

• smooth ECE (primary) — range: [0, 1]
  • Expected Calibration Error measuring the difference between predicted confidence and actual accuracy, computed using smoothed bin boundaries to reduce variance.
• binned ECE — range: [0, 1]
  • Standard ECE computed over a fixed number of equal-width bins (B∈{10,15,20}).
• Brier score — range: [0, 1]
  • Mean squared difference between predicted probability and actual outcome (1 if correct, 0 otherwise).
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, used to evaluate selective prediction performance.

Input / output format

Input: Tabular data presented in a specific serialization format (e.g., Markdown, HTML, JSON, CSV) alongside a natural language question.

Output: A predicted answer string and a scalar confidence score (probability) indicating the model's certainty in the answer.

Scoring recipe

def compute_ece(confidences, accuracies, n_bins=15):
    bin_boundaries = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    for i in range(n_bins):
        mask = (confidences >= bin_boundaries[i]) & (confidences < bin_boundaries[i+1])
        if mask.sum() > 0:
            bin_acc = accuracies[mask].mean()
            bin_conf = confidences[mask].mean()
            ece += mask.sum() * abs(bin_acc - bin_conf)
    return ece / len(confidences)

def compute_brier(confidences, accuracies):
    return np.mean((confidences - accuracies) ** 2)

Common pitfalls

• Strict string matching fails on formatting variations (e.g., '37 women competed' vs '37'), requiring a strict-then-fuzzy pipeline.
• Confidence scores are highly sensitive to table serialization format, leading to systematic overconfidence if not perturbed.
• Bootstrap CIs are only reported for TableBench results, not WTQ.

Evidence (verbatim from paper)

Metrics: smooth ECE (Błasiok and Nakkiran, 2024), binned ECE (B∈{10,15,20}), Brier score, and AUROC for selective prediction. A strict-then-fuzzy matching pipeline handles formatting differences (e.g., “37 women competed” vs gold “37”).

Citation

@misc{voss2026calibrated,
  title={Calibrated Confidence Estimation for Tabular Question Answering},
  author={Voss (2026)},
  year={2026},
  note={arXiv:2604.12491}
}

• arXiv: 2604.12491