# Latentrefusal Eval

> Evaluates a model's ability to detect unanswerable Text-to-SQL queries by analyzing intermediate hidden activations, aiming to prevent hallucinated SQL generation and unsafe execution. It probes whether the system can reliably distinguish between answerable and unanswerable prompts across diverse domains and linguistic ambiguities. Use when the user wants to benchmark on TriageSQL, AMBROSIA, SQuAD 2.0, MD-Enterprise, or asks about evaluating this task. Reports F1.

- Skill: `qhjqhj00/latentrefusal-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/latentrefusal-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/latentrefusal-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/latentrefusal-eval

---


# latentrefusal-eval

> LatentRefusal: Latent-Signal Refusal for Unanswerable Text-to-SQL Queries — Ren et al. (2026) (arXiv:2601.10398, 2026)

## What this evaluates

Evaluates a model's ability to detect unanswerable Text-to-SQL queries by analyzing intermediate hidden activations, aiming to prevent hallucinated SQL generation and unsafe execution. It probes whether the system can reliably distinguish between answerable and unanswerable prompts across diverse domains and linguistic ambiguities.

## Datasets

- **TriageSQL** — total ?; splits: test (-1)
- **AMBROSIA** — total ?; splits: test (-1)
- **SQuAD 2.0** — total ?; splits: test (-1)
- **MD-Enterprise** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: percent
  - F1 score computed at a fixed decision threshold tuned on development data. It balances precision and recall for binary refusal detection.

## Input / output format

**Input**: Natural language question paired with a database schema (Text-to-SQL prompt).

**Output**: Binary classification: 'answerable' or 'unanswerable' (refusal decision).

## Scoring recipe

```python
def compute_f1(predictions, gold_labels, threshold=0.5):
    preds_bin = [1 if p >= threshold else 0 for p in predictions]
    tp = sum(1 for p, g in zip(preds_bin, gold_labels) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds_bin, gold_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds_bin, gold_labels) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) * 100
```

## Common pitfalls

- Threshold for refusal decision is fixed and tuned on development data, not automatically optimized per test split.
- Sampling-based baselines require multiple forward passes, whereas LatentRefusal uses a single greedy pass, making latency comparisons unfair if not accounted for.
- Syntactic diversity in SQL generation can inflate spectral uncertainty metrics, causing them to misinterpret confident hallucinations as uncertainty.

## Evidence (verbatim from paper)

> Table 1 summarizes the refusal detection performance across four benchmarks. We report F1 as the primary metric, computed at a fixed decision threshold tuned on development data.

## Citation

```bibtex
@misc{ren2026latentrefusal,
  title={LatentRefusal: Latent-Signal Refusal for Unanswerable Text-to-SQL Queries},
  author={Ren et al. (2026)},
  year={2026},
  note={arXiv:2601.10398}
}
```

- arXiv: 2601.10398