Finsquad Eval

finsquad-eval

Finnish SQuAD: A Simple Approach to Machine Translation of Span Annotations — Nuutinen et al. (2025) (arXiv:2501.05963, 2025)

What this evaluates

Evaluates the quality of a machine-translated extractive QA dataset (FinSQuAD) by training and testing QA models on it, comparing performance against other translated SQuAD datasets and the original English version. It also assesses translation fidelity through backtranslation and manual error analysis.

Datasets

• Finnish SQuAD2.0 — total ?; splits: train (-1), val (-1), test (-1)
• SQuAD2.0 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• exact match (EM) (primary) — range: [0, 1]
  • The proportion of questions that receive the exactly correct answer span. A prediction matches the gold span exactly.
• token F1 — range: [0, 1]
  • The F1 score of the precision and recall of tokens in the predicted answer span compared to the reference answer span. More tolerant to minor changes at the span boundaries.

Input / output format

Input: A question and a context passage in Finnish (or English for baseline models).

Output: A predicted answer span (text string) extracted from the context.

Scoring recipe

def compute_metrics(predictions, references):
    em_scores = []
    f1_scores = []
    for pred, ref in zip(predictions, references):
        pred_tokens = normalize(pred)
        ref_tokens = normalize(ref)
        em_scores.append(1.0 if pred_tokens == ref_tokens else 0.0)
        if not pred_tokens or not ref_tokens:
            f1_scores.append(0.0)
            continue
        common = Counter(pred_tokens) & Counter(ref_tokens)
        num_same = sum(common.values())
        precision = num_same / len(pred_tokens)
        recall = num_same / len(ref_tokens)
        f1 = (2 * precision * recall) / (precision + recall)
        f1_scores.append(f1)
    return {'EM': sum(em_scores) / len(em_scores), 'F1': sum(f1_scores) / len(f1_scores)}

Common pitfalls

• Assuming the performance drop on the translated dataset is solely due to translation noise; it also includes inherent language and model architecture differences.
• Interpreting backtranslation evaluation results as direct single-round translation error; backtranslation accumulates errors over two translation rounds, so the reported drop overestimates the actual impact on the target model.
• Confusing token F1 with exact match; F1 measures token overlap and is tolerant to boundary shifts, but does not guarantee exact span recovery.

Evidence (verbatim from paper)

We use as metrics the exact match (EM), the proportion of questions that receive the exactly correct answer span, and token F1, the F1 score of the precision and recall of tokens in the predicted answer span, compared to the reference answer span. The latter metric is more tolerant to minor changes at the span boundaries.

Citation

@misc{nuutinen2025finsquad,
  title={Finnish SQuAD: A Simple Approach to Machine Translation of Span Annotations},
  author={Nuutinen et al. (2025)},
  year={2025},
  note={arXiv:2501.05963}
}

• arXiv: 2501.05963