Structured Prompting Eval

structured-prompting-eval

Structured Prompting Enables More Robust Evaluation of Language Models — Aali et al. (2025) (arXiv:2511.20836, 2025)

What this evaluates

Evaluates how different prompting strategies (baseline, zero-shot, chain-of-thought, and automated optimizers) affect the accuracy, ranking stability, and variance of language model performance across multiple knowledge and reasoning benchmarks.

Datasets

• MMLU-Pro — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• MedCalc-Bench — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)
• HeadQA — total ?; splits: test (-1)
• MedBullets — total ?; splits: test (-1)
• Medec — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers out of the total number of instances. Macro-averaged across benchmarks when reporting aggregate performance.

Input / output format

Input: Task-specific question or prompt (e.g., multiple-choice question, patient note) prepended with a structured prompt template containing instructions and/or few-shot demonstrations.

Output: Final answer string, optionally preceded by a chain-of-thought reasoning trace. Output length capped at <200 tokens.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
    return (correct / len(gold)) * 100

def compute_mean_rank(model_accuracies):
    ranks = sorted(model_accuracies.items(), key=lambda x: x[1], reverse=True)
    return {name: rank + 1 for rank, (name, _) in enumerate(ranks)}

Common pitfalls

• Prompt design significantly alters leaderboard rankings; a model's rank is not invariant to the prompting method used.
• Non-reasoning baselines (Zero-Shot Predict) underperform by ~4% on average compared to CoT variants, which can mask true capability differences.
• Optimization costs (BFRS, MIPROv2) are one-time amortized expenses, while inference costs scale with prompt token length.

Evidence (verbatim from paper)

Structured prompting methods (Zero-Shot CoT, BFRS, MIPROv2) consistently improve over the HELM baseline (Table [3]). On average, LMs gain $+$4% in absolute accuracy.

Citation

@misc{aali2025structuredprompting,
  title={Structured Prompting Enables More Robust Evaluation of Language Models},
  author={Aali et al. (2025)},
  year={2025},
  note={arXiv:2511.20836}
}

• arXiv: 2511.20836