R2pe Eval

r2pe-eval

Can We Verify Step by Step for Incorrect Answer Detection? — Xu et al. (2024) (arXiv:2402.10528, 2024)

What this evaluates

Evaluates the ability to detect incorrect answers in chain-of-thought reasoning by analyzing inconsistencies across multiple reasoning paths. It measures how well intermediate steps can predict the correctness of a final answer without relying solely on the answer itself.

Datasets

• R2PE — total 38300; splits: full (38300); repo https://github.com/XinXU-USTC/R2PE

Metrics

• Discernibility Score (DS) (primary) — range: [0, 1]
  • A numerical criterion that encapsulates the quality of rationales and answers across multiple reasoning paths. Predictions are made by comparing the DS against a threshold H (if DS < H, predict FALSE; else TRUE). Detection performance is subsequently evaluated using F1 score.

Input / output format

Input: Question or claim Q, dataset name, queried LLM name, five CoT responses each containing a rationale r_i and extracted answer a_i, the final aggregated answer a, the ground-truth answer, and the ground-truth label L (TRUE/FALSE).

Output: Predicted label ̂L ∈ {TRUE, FALSE} based on whether the Discernibility Score (DS) falls below a threshold H.

Scoring recipe

predictions = []
for instance in dataset:
    ds = compute_discernibility_score(instance.responses)
    pred = 'FALSE' if ds < threshold_H else 'TRUE'
    predictions.append(pred)
tp = sum(1 for p, g in zip(predictions, dataset.labels) if p == 'TRUE' and g == 'TRUE')
fp = sum(1 for p, g in zip(predictions, dataset.labels) if p == 'TRUE' and g == 'FALSE')
fn = sum(1 for p, g in zip(predictions, dataset.labels) if p == 'FALSE' and g == 'TRUE')
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• The exact mathematical formula for the Discernibility Score (DS) is not provided in the benchmark construction section and must be retrieved from the paper's methodology.
• The benchmark aggregates answers via majority vote across 5 CoT generations per question before labeling, which can mask individual reasoning path errors.
• Evaluation is stratified into 45 subsets based on the combination of source dataset and generating LLM, so reporting aggregate scores without subset breakdown may hide significant performance variations.

Evidence (verbatim from paper)

We introduce a numerical criterion, the discernibility score (DS), to encapsulate the quality of the rationales and their corresponding answers. A low DS might suggest a potential mismatch between the final output a and the ground-truth answer. Hence, we will classify the example as false: ̂L=F, if its DS falls below a certain threshold H.

Citation

@misc{xu2024r2pe,
  title={Can We Verify Step by Step for Incorrect Answer Detection?},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2402.10528}
}

• arXiv: 2402.10528