Multi Prompt LLM Eval

multi-prompt-llm-eval

State of What Art? A Call for Multi-Prompt LLM Evaluation — Moran Mizrahi et al. (2024) (arXiv:2401.00595, 2024)

What this evaluates

This evaluation protocol probes the robustness of large language models to instruction phrasing by measuring performance across multiple semantically equivalent prompts. It assesses whether model rankings and absolute scores remain stable when the same task is presented with different instruction templates.

Datasets

• LMentry — total ?; splits: test (-1)
• BIG-bench Lite — total ?; splits: test (-1)
• BIG-bench Hard — total ?; splits: test (-1)

Metrics

• exact match evaluation (primary) — range: [0, 1]
  • Compares the model's generated output string directly to the ground-truth label string. A match yields 1, otherwise 0. Averaged over all instances in the benchmark.

Input / output format

Input: A task instruction (provided in one of several paraphrased templates for LMentry, or a single fixed template for BIG-bench tasks) followed by the task-specific prompt or question.

Output: The model's generated text response, which is compared verbatim to the expected answer.

Scoring recipe

def score(predictions, golds):
    matches = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip() == gold.strip():
            matches += 1
    return matches / len(golds)

Common pitfalls

• Single-prompt evaluations are highly brittle and can misrepresent model capabilities due to instruction phrasing sensitivity.
• Closed API models may manipulate or wrap input prompts, interfering with direct evaluation of instruction templates.
• Exact-match scoring is strict and may penalize semantically correct but syntactically different model outputs.

Evidence (verbatim from paper)

We measure performance in the standard manner provided by each benchmark. In LMentry this is done with the official evaluation script, while in Big-Bench we use exact match evaluation. We note that while this evaluation is somewhat strict, we believe that it is also fair and straightforward.

Citation

@misc{mizrahi2024multiprompt,
  title={State of What Art? A Call for Multi-Prompt LLM Evaluation},
  author={Moran Mizrahi et al. (2024)},
  year={2024},
  note={arXiv:2401.00595}
}

• arXiv: 2401.00595