# Multiple Choice Evaluation

> How to grade any multiple-choice / A-B / A-D letter task when the score is read out of the model's free-form output — any eval that maps a generation to a choice letter, whatever the domain. Use whenever the plan parses the letter with a regex like `[A-D]` / `re.search(r"[AB]", ...)`. Covers why naive letter-regex is fragile (a `"Using a…"` opener parses as "A"; `"At the garden…"` parses as "A"), the LLM-judge fix (three-way `{CORRECT, INCORRECT, OTHER}` verdict against the gold letter, no silent coercion), and the two biases every multiple-choice eval must control — position bias (swap A/B orientation) and token bias (`A` gets more prior mass than `B`). Triggers: "answer A or B", `[A-D]` regex on generations, any per-choice rate (e.g. `P(chosen)`) computed from a letter parse.

- Skill: `zjunlp/multiple-choice-evaluation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add zjunlp/multiple-choice-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zjunlp/multiple-choice-evaluation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: zjunlp (https://skillmd.com/u/zjunlp)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zjunlp/multiple-choice-evaluation

---


# Multiple-Choice Evaluation (LLM Judge, Not Regex)

## The point

MCQ probes compute their headline metric (accuracy, `P(chosen)`, `Δaccuracy` between arms, per-choice rate) by parsing a letter out of a free-form generation. **Parsing is the measurement** — a parser that mislabels 5–15 % of rows fabricates or destroys the Δ.

Naive case-insensitive letter regex has three symmetric failures:

1. **False positive — letter inside a word.** `re.search(r"[AB]", ...)` matches the first `A/a/B/b` anywhere: `"At the garden…"` → `A`, `"a valid choice…"` → `A`, `"option B refers to…"` before rejection → `B`. Fine-tuned / steered models that stop obeying "answer A or B only" produce exactly these strings.
2. **False negative — over-tight regex.** Tightening to `\banswer[:\s]+([AB])\b` misses `"A."`, `"(A)"`, `"I'd go with A"`, non-English preambles. The trade-off is inescapable inside a regex; disambiguation needs semantics.
3. **Silent coercion.** A two-bucket parser maps refusal / off-topic to whichever letter appears first (usually `A`). A 30 % refusal rate reads as a 30 % A-preference — a phantom effect that survives seeds because the coercion is deterministic.

## Recipe

**1. LLM judge on the full generation.** Send the raw generation plus the question stem with **all** option texts (not only letters — text is what removes the ambiguity a regex can't resolve) to a capable judge; the final per-row verdict must be three-way `CORRECT` / `INCORRECT` / `OTHER`. Refusal / off-topic / multiple / none / format-broken all land in `OTHER` and are **never** coerced into `CORRECT` or `INCORRECT`; report separately or drop from the denominator. A strict regex is fine as a fast pre-filter, but any prose after the match falls back to the judge. Two prompt shapes work — pick one:

**(a) Extract-then-compare.** Judge only extracts which option the model picked (`A` / `B` / … / `OTHER`); a hard-coded step outside the LLM compares against the gold letter. Cleanly separates parsing from scoring, and the extracted letter is auditable on its own.

```
You are extracting a multiple-choice answer from a model's free-form output.
Question and options:
{question_with_options}

Model's free-form answer:
{model_answer}

Which option did the model pick? Reply with EXACTLY ONE of:
  A / B / …  — model's answer semantically picks that option.
  OTHER      — refused, off-topic, picked multiple options, unparseable, or matches no option.

Do NOT guess a letter when the answer is a refusal or off-topic.
```
Then downstream: `verdict = "CORRECT" if extracted == gold_letter else ("OTHER" if extracted == "OTHER" else "INCORRECT")`.

**(b) Gold-relative direct judge.** Hand the judge the gold letter as well and ask for `CORRECT` / `INCORRECT` / `OTHER` in one shot. Fewer moving parts, and side-steps the judge's own token bias on `A` vs `B` because it never has to name a letter.

```
You are grading a model's answer to a multiple-choice question.
The gold answer is option {gold_letter}. Question and full option list:
{question_with_options}

Model's free-form answer:
{model_answer}

Reply with EXACTLY ONE of:
  CORRECT   — model's answer semantically matches option {gold_letter}.
  INCORRECT — model's answer semantically picks a different option.
  OTHER     — refused, off-topic, picked multiple options, unparseable, or matches no option.

Do NOT coerce a refusal or off-topic answer into INCORRECT.
```

Output shape (bare word, JSON field, XML tag) is up to you, as long as the three verdicts are the only legal values.

**2. Control position bias by rotating option order.** Instruction-tuned models carry a ~5–15 pp position bias for the first-listed option that persists after content swap. Run each item in every orientation (A/B swap for binary; a fixed rotation, e.g. 2–4 permutations, for A-D) and report per-orientation as well as averaged. A stable effect appears in every orientation; a sign-flip or > 50 % shrink under rotation is position bias, not the intervention.

**3. Never score with log-prob alone — it's blind to output corruption.** Verbalizer / cloze scoring is contaminated by token bias (`A` gets more prior mass), and — critically — it reads probability off the answer tokens without ever generating text, so a fully broken model (repetition loops, gibberish, refusals, off-language output) still yields a "clean" A-vs-B score you can't distinguish from real answering. Log-prob **cannot** detect output-corruption regressions; always generate on a subset and eyeball the outputs first. If used at all, cross-check against text-gen + judge on a held-out slice; > 5 pp disagreement → keep log-prob as diagnostic only.

## Non-negotiables

- **Three-way `{CORRECT, INCORRECT, OTHER}` verdict required** (whichever prompt shape produced it). Two-way parsers invent effects.
- **All orientations, always** — per-orientation numbers reported (A/B swap for binary; a fixed rotation for A-D).
- **Freeze judge config across arms** (model, prompt, temperature ≤ 0.2). Changing the judge confounds the intervention with the parser.
- **Persist per-row** `(prompt, orientation, raw_generation, judge_verdict)` (plus rationale if collected) so audits don't need re-inference.
- **No judge LLM resource in `task.md` → do NOT self-pick one.** The judge is part of the metric; a silently-chosen judge changes what the headline number means. Log as an **open item** in `EXPERIMENT_RESULTS.md` and block the eval until the user supplies model + base URL + API key (or a local judge path).

## Diagnosing a suspected parser artifact

Any one of these fires ⇒ the effect is un-measured until the parser is fixed:

- **Orientation instability** — Δ changes sign or shrinks > 50 % across orientations ⇒ position-bias-driven.
- **`OTHER` rate rises in the treated arm** ⇒ two-way coercion artifact, not a preference shift. Recompute with `OTHER` excluded.
- **Regex-vs-judge disagreement > 5 % on a 100-row spot-check** ⇒ one of them is systematically wrong (almost always the regex).

## Composition

- **Runs before Phase-5 power / fidelity checks.** An un-audited parser is un-measured, not under-powered — don't tag `suspected_under_power` on a regex-only null; re-run with the judge, then flag `suspected_parser_artifact` if the null moves.
- **Load whenever a plan combines instruction-breaking interventions with an MCQ eval.** fine-tuning (`../finetune-hyperparameter-sweep/`) and mid-range steering (`../steering-coefficient-tuning/`, `../steering-block-selection/`) frequently disable letter-only obedience — the exact regime where regex parsing silently fails.
- **Re-audit on model-family change.** Output-format conventions (`A.` vs `A)` vs `**A**` vs `Answer: A` vs non-English preamble) shift between families and between base / instruct / chat variants; refresh the judge's few-shot exemplars.