Majority K

majority-k

RAST: Reasoning Activation in LLMs via Small-model Transfer — Ouyang et al. (2025) (arXiv:2506.15710, 2025)

What this evaluates

Evaluates the robustness and consistency of a model's mathematical reasoning by aggregating predictions across multiple inference runs per problem. It measures whether the most frequent prediction among k samples matches the ground truth.

Datasets

• MATH500 — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• AIME 2024 — total ?; splits: test (-1)
• AMC 2023 — total ?; splits: test (-1)
• Minerva — total ?; splits: test (-1)
• Olympiad Bench — total ?; splits: test (-1)

Metrics

• Majority@k (primary) — range: [0, 1]
  • Accuracy based on the majority prediction among k inference runs per problem. Formula: (1/N) * sum_{i=1}^N I(majority(y_hat_{i,1}...y_hat_{i,k}) == y_i).

Input / output format

Input: Mathematical problem statement.

Output: A predicted answer string per inference run.

Scoring recipe

def compute_majority_k(predictions_list, k, ground_truths):
    correct = 0
    for preds, gt in zip(predictions_list, ground_truths):
        majority_pred = max(set(preds), key=preds.count)
        if majority_pred == gt:
            correct += 1
    return correct / len(ground_truths)

Common pitfalls

• Requires generating k independent samples per problem, which significantly increases inference cost.
• Tie-breaking in the majority function is not specified and can arbitrarily affect results when k is small.
• Accuracy is typically exact-match on the final answer, not step-level or format-tolerant.

Evidence (verbatim from paper)

Definition of Majority@k: This metric evaluates accuracy based on the majority prediction among multiple inference runs per problem. Formally, Majority@k = 1/N \sum_{i=1}^{N}\mathbb{I}(majority(\hat{y}{i,1},\dots,\hat{y}{i,k}) = y_i), where the majority function returns the prediction most frequently appearing among the k inference runs for the i-th problem.

Citation

@misc{ouyang2025rast,
  title={RAST: Reasoning Activation in LLMs via Small-model Transfer},
  author={Ouyang et al. (2025)},
  year={2025},
  note={arXiv:2506.15710}
}

• arXiv: 2506.15710