Heq Eval

heq-eval

HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark — Amir DN Cohen et al. (arXiv:2508.01812, 2025)

What this evaluates

This benchmark evaluates extractive reading comprehension in Hebrew, a morphologically rich language. It probes a model's ability to accurately identify answer spans within a given context passage, handling challenges like affixation, spelling variations, and domain-specific vocabulary across news and encyclopedic text.

Datasets

• HeQ — total 30147; splits: train (-1), test (-1); repo https://github.com/NNLP-IL/Hebrew-Question-Answering-Dataset

Metrics

• EM — range: [0, 1]
  • Exact Match accuracy. Returns 1 if the predicted answer span exactly matches the gold answer span character-for-character, and 0 otherwise.
• F1 — range: [0, 1]
  • Token-level F1 score. Computes the harmonic mean of precision and recall based on overlapping tokens between the predicted and gold answer spans.
• TLNLS (primary) — range: [0, 1]
  • Token-Level Normalized Levenshtein Similarity. Computes the normalized Levenshtein distance between the tokenized predicted and gold answers, then subtracts it from 1 to yield a similarity score that tolerates morphological variations and minor span boundary shifts.

Input / output format

Input: A context passage (in Hebrew) and a corresponding question.

Output: An extracted answer span from the context, or an empty string if the answer is not present in the text.

Scoring recipe

def compute_metrics(predictions, golds):
    em, f1, tlnls = [], [], []
    for pred, gold in zip(predictions, golds):
        pred, gold = pred.strip(), gold.strip()
        em.append(1.0 if pred == gold else 0.0)
        pred_tokens, gold_tokens = tokenize(pred), tokenize(gold)
        # F1 calculation omitted for brevity
        # TLNLS = 1 - (LevenshteinDistance(pred_tokens, gold_tokens) / max(len(pred_tokens), len(gold_tokens)))
        tlnls.append(1.0 - (levenshtein(pred_tokens, gold_tokens) / max(len(pred_tokens), len(gold_tokens))))
    return {'EM': sum(em)/len(em), 'F1': sum(f1)/len(f1), 'TLNLS': sum(tlnls)/len(tlnls)}

Common pitfalls

• Span boundary inaccuracies: Models frequently return correct answers but with incorrect start/end indices, which heavily penalizes EM despite semantic correctness.
• Morphological sensitivity: Standard character-level exact match fails to account for Hebrew affixation and orthographic variations, leading to artificially low scores without token-level or normalized similarity metrics.
• Domain shift masking: Aggregate scores often hide significant performance drops when models trained on one domain (e.g., Wikipedia) are evaluated on another (e.g., tech news).

Evidence (verbatim from paper)

We evaluate several known pre-trained models on the HeQtest set. Table 4: Performance Comparison of Different Models on the HeQDataset. ... | Model | EM | $F_{1}$ | TLNLS |

Citation

@misc{cohen2025heq,
  title={HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark},
  author={Amir DN Cohen et al.},
  year={2025},
  note={arXiv:2508.01812}
}

• arXiv: 2508.01812