Latentrefusal Eval

latentrefusal-eval

LatentRefusal: Latent-Signal Refusal for Unanswerable Text-to-SQL Queries — Ren et al. (2026) (arXiv:2601.10398, 2026)

What this evaluates

Evaluates a model's ability to detect unanswerable Text-to-SQL queries by analyzing intermediate hidden activations, aiming to prevent hallucinated SQL generation and unsafe execution. It probes whether the system can reliably distinguish between answerable and unanswerable prompts across diverse domains and linguistic ambiguities.

Datasets

• TriageSQL — total ?; splits: test (-1)
• AMBROSIA — total ?; splits: test (-1)
• SQuAD 2.0 — total ?; splits: test (-1)
• MD-Enterprise — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: percent
  • F1 score computed at a fixed decision threshold tuned on development data. It balances precision and recall for binary refusal detection.

Input / output format

Input: Natural language question paired with a database schema (Text-to-SQL prompt).

Output: Binary classification: 'answerable' or 'unanswerable' (refusal decision).

Scoring recipe

def compute_f1(predictions, gold_labels, threshold=0.5):
    preds_bin = [1 if p >= threshold else 0 for p in predictions]
    tp = sum(1 for p, g in zip(preds_bin, gold_labels) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds_bin, gold_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds_bin, gold_labels) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) * 100

Common pitfalls

• Threshold for refusal decision is fixed and tuned on development data, not automatically optimized per test split.
• Sampling-based baselines require multiple forward passes, whereas LatentRefusal uses a single greedy pass, making latency comparisons unfair if not accounted for.
• Syntactic diversity in SQL generation can inflate spectral uncertainty metrics, causing them to misinterpret confident hallucinations as uncertainty.

Evidence (verbatim from paper)

Table 1 summarizes the refusal detection performance across four benchmarks. We report F1 as the primary metric, computed at a fixed decision threshold tuned on development data.

Citation

@misc{ren2026latentrefusal,
  title={LatentRefusal: Latent-Signal Refusal for Unanswerable Text-to-SQL Queries},
  author={Ren et al. (2026)},
  year={2026},
  note={arXiv:2601.10398}
}

• arXiv: 2601.10398