Respondeoqa Eval

respondeoqa-eval

RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering — Hudspeth et al. (2026) (arXiv:2604.20738, 2026)

What this evaluates

This benchmark evaluates large language models on bilingual Latin-English question answering across knowledge-based, skill-based (grammar, scansion, literary devices), multihop reasoning, and translation tasks. It probes models' ability to handle classical language morphology, poetic meter analysis, and cross-lingual generation under constrained and unconstrained settings.

Datasets

• RespondeoQA — total 7800; splits: test (-1); repo https://github.com/slanglab/RespondeoQA

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • 1 if the normalized predicted answer matches the normalized gold answer, else 0. Normalization includes lowercasing, stripping punctuation/whitespace, replacing J with I, V with U, and normalizing macrons/accents/ligatures. Averaged over all MC and 1-word SA instances.
• mean per-item accuracy — range: [0, 1]
  • For scansion feet identification: the fraction of correctly identified metrical feet across all items in the dataset. Partial correctness is allowed per foot.
• accuracy — range: [0, 1]
  • For scansion meter manipulation: 1 if the moved word is placed in the correct position to form valid pentameter/hexameter, else 0. Averaged over instances.
• BLEU score — range: percent
  • sacreBLEU implementation for long-form translation evaluation. Measures n-gram overlap between model output and reference translations.

Input / output format

Input: System prompt: 'You are a Classicist with expert knowledge in Greek and Roman history, language, and culture.' followed by the question text (multiple choice, 1-word short answer, scansion line, or translation prompt).

Output: For MC: ends with the letter of the correct answer. For 1-word SA: ends with a single word. For scansion/translation: free-form text following specific prompt instructions.

Scoring recipe

def normalize(text):
    return text.lower().strip().replace('j','i').replace('v','u')
def score_mc_or_sa(pred, gold):
    return 1.0 if normalize(pred) == normalize(gold) else 0.0
def score_scansion_feet(pred, gold):
    pred_f, gold_f = parse_feet(pred), parse_feet(gold)
    return sum(p==g for p,g in zip(pred_f, gold_f)) / len(gold_f) if gold_f else 0.0
def score_scansion_meter(pred, gold):
    return 1.0 if pred == gold else 0.0
def score_translation(pred, gold):
    return sacrebleu.corpus_bleu([pred], [[gold]]).score

Common pitfalls

• Latin orthography normalization (I/J, U/V, macrons) must be applied before exact match; otherwise scores are artificially low.
• Constrained translation prompts significantly alter the answer space and BLEU scores compared to unconstrained prompts.
• Multihop question classification is subjective and may not align with human reasoning steps, affecting performance analysis.

Evidence (verbatim from paper)

For multiple choice questions, we evaluate the accuracy of the predicted letter choice. For 1-word short answer questions, we use exact match (EM) accuracy, after normalization (lowercasing, stripping punctuation and whitespace, JV replacement, normalizing macrons, accents, and ligatures). For a subset of scansion questions ("feet identification"), we use mean per-item accuracy (see row 10 of Table 2 for an example). Each question gives one line of poetry and asks for the name of each metrical foot in the order it appears in the verse. For a single line, the number of feet typically ranges from 3-6, and partial correctness is allowed. For another subset of scansion questions ("meter manipulation"), we use accuracy (see row 11 of Table 2). Each question gives a line of poetry and asks that the position of a single word be changed in order to make the verse valid pentameter or hexameter. Since only one word should be moved, we consider it correct (1) if the word is in the correct position and incorrect (0) otherwise. Finally, for long-form translation questions, we report the BLEU score Papineni et al. (2002).

Citation

@misc{hudspeth2026respondeoqa,
  title={RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering},
  author={Hudspeth et al. (2026)},
  year={2026},
  note={arXiv:2604.20738}
}

• arXiv: 2604.20738