Mevaker Conclusion Eval

mevaker-conclusion-eval

Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Language — Shalumov et al. (2024) (arXiv:2403.09719, 2024)

What this evaluates

Evaluates a model's ability to perform binary classification for identifying conclusion sentences in Hebrew audit reports, and to rank sentence pairs by semantic similarity for hierarchical conclusion allocation.

Datasets

• MevakerConcSen — total ?; splits: test (-1)
• PS (Parallel Sentences) — total 999000; splits: dev (1000)

Metrics

• F1 (primary) — range: [0, 1]
  • Standard harmonic mean of precision and recall for binary sentence-level classification.
• Kendall Rank Correlation (KRC) (primary) — range: [-1, 1]
  • Kendall's rank correlation coefficient measuring the correspondence between predicted and ground truth similarity rankings.
• Mean Absolute Error (MAE) — range: [0, 1]
  • Mean absolute difference between predicted and ground truth similarity scores, normalized by the number of samples.

Input / output format

Input: For extraction: a window of N sentences (context + target). For allocation: a pair of sentences.

Output: For extraction: binary label (conclusion vs. non-conclusion). For allocation: similarity score used to rank sentence pairs.

Scoring recipe

# F1 for extraction
tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

# KRC & MAE for allocation
krc = kendalltau(preds, gold).correlation
mae = mean(abs(np.array(preds) - np.array(gold))) / len(gold)

Common pitfalls

• Sliding window training requires disabling shuffling to prevent train/test leakage from overlapping windows.
• Evaluation prioritizes relative ranking metrics (KRC) over absolute similarity scores to ensure fair comparison between cross-encoders and bi-encoders.
• Training splits are artificially balanced via duplication, but test splits retain the original severe class imbalance.

Evidence (verbatim from paper)

The evaluation was performed on two metrics - Kendall Rank Correlation (KRC) (Kendall, 1938) and Mean Absolute Error (MAE) normalized by number of samples.

Citation

@misc{shalumov2024mevaker,
  title={Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Language},
  author={Shalumov et al. (2024)},
  year={2024},
  note={arXiv:2403.09719}
}

• arXiv: 2403.09719