Multiple Choice Evaluation

How to grade any multiple-choice / A-B / A-D letter task when the score is read out of the model's free-form output — any eval that maps a generation to a choice letter, whatever the domain. Use whenever the plan parses the letter with a regex like `[A-D]` / `re.search(r"[AB]", ...)`. Covers why naive letter-regex is fragile (a `"Using a…"` opener parses as "A"; `"At the garden…"` parses as "A"), the LLM-judge fix (three-way `{CORRECT, INCORRECT, OTHER}` verdict against the gold letter, no silent coercion), and the two biases every multiple-choice eval must control — position bias (swap A/B orientation) and token bias (`A` gets more prior mass than `B`). Triggers: "answer A or B", `[A-D]` regex on generations, any per-choice rate (e.g. `P(chosen)`) computed from a letter parse.

zjunlp Updated 1.5k repo stars

File contents

zjunlp/mechanist commit 2dddea992c

Frequently asked questions

npx get-skill add zjunlp/multiple-choice-evaluation