Decap Bbq Eval

decap-bbq-eval

DeCAP: Context-Adaptive Prompt Generation for Debiasing Zero-shot Question Answering in Large Language Models — Bae et al. (2025) (arXiv:2503.19426, 2025)

What this evaluates

Evaluates zero-shot question answering robustness against social biases, specifically testing how models adapt to ambiguous versus unambiguous contexts without relying on internal stereotypical knowledge.

Datasets

• BBQ — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified question types (ambiguous vs. unambiguous) on the BBQ dataset.

Input / output format

Input: Context and question provided to the LLM, optionally augmented with a prefix instruction and few-shot neutral answer guidance demonstrations.

Output: A generated answer with reasoning, or a classification label (ambiguous/unambiguous) based on context similarity.

Scoring recipe

# Ambiguity detection scoring
answer_with_reason = llm.generate(context, question, prompt="answer with reason")
rouge_score = compute_rouge(context, answer_with_reason)
is_ambiguous = rouge_score < threshold
accuracy = mean(is_ambiguous == ground_truth_label)

Common pitfalls

• Relying on LLM internal knowledge for ambiguity detection causes significant performance gaps between ambiguous and unambiguous questions.
• Fixed or self-reliant debiasing prompts degrade performance specifically in ambiguous contexts where context information is insufficient.

Evidence (verbatim from paper)

Table[2]: Experimental results for the question ambiguity detector on the BBQ dataset. The table shows the accuracy (%) in correctly classifying each question type. ... To quantify this similarity, in the second step, we calculate the ROUGE score*(Lin, [2004])* between the context and the answer combined with the question. If the score is lower than our threshold, the question is classified as ambiguous; if the score is higher, the question is classified as unambiguous.

Citation

@misc{bae2025decap,
  title={DeCAP: Context-Adaptive Prompt Generation for Debiasing Zero-shot Question Answering in Large Language Models},
  author={Bae et al. (2025)},
  year={2025},
  note={arXiv:2503.19426}
}

• arXiv: 2503.19426