Parsinlu Eval

parsinlu-eval

ParsiNLU: A Suite of Language Understanding Challenges for Persian — Khashabi et al. (2020) (arXiv:2012.06154, 2020)

What this evaluates

Evaluates Persian language understanding across six distinct NLU tasks, including reading comprehension, textual entailment, sentiment analysis, and machine translation. It measures how well pre-trained monolingual and multilingual models perform on native-speaker annotated Persian data compared to human baselines.

Datasets

• ParsiNLU — total ?; splits: train (-1), dev (-1), eval (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Exact match F1 score computed between the predicted answer string and the gold answer string for reading comprehension tasks.
• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or answers across classification tasks such as paraphrasing, entailment, multiple-choice QA, and aspect-specific sentiment.
• Macro-F1 — range: [0, 1]
  • Macro-averaged F1 score computed across all classes, treating each class equally, used for document-level sentiment and aspect extraction.
• Sacre-BLEU — range: [0, 100]
  • BLEU score computed using the SacreBLEU toolkit with standard tokenization and sentence-level scoring for machine translation evaluation.

Input / output format

Input: Varies by task: (1) Reading comprehension: context passage + question; (2) Classification tasks (entailment, paraphrasing, MCQA, sentiment): input text(s) formatted as questions or pairs; (3) Machine translation: source Persian sentence.

Output: Varies by task: (1) Reading comprehension: predicted answer string; (2) Classification tasks: predicted label or sentiment polarity; (3) Machine translation: translated Persian sentence.

Scoring recipe

if task == 'reading_comprehension':
    score = f1_score(gold_answer, pred_answer)
elif task in ['paraphrasing', 'entailment', 'mcqa', 'aspect_sentiment']:
    score = accuracy_score(gold_label, pred_label)
elif task in ['doc_sentiment', 'aspect_extraction']:
    score = macro_f1_score(gold_labels, pred_labels)
elif task == 'machine_translation':
    score = sacrebleu.compute([pred_translation], [gold_translation])

Common pitfalls

• Human performance is estimated on small random subsets (100-150 instances) per task rather than the full evaluation set.
• English datasets cannot be directly used for sentiment analysis due to schema incompatibility, requiring task-specific Persian data.
• Model selection is strictly based on development set performance, not test set performance.

Evidence (verbatim from paper)

For reading-comprehension, we use $F1$ between gold answer and the response string (Rajpurkar et al., 2016); for question paraphrasing, textual entailment, multiple-choice question-answering, and sentiment analysis, we use accuracy. For the first two sub-tasks of sentiment analysis (document-level sentiment, aspect extraction), we use macro- $F1$ . For the third sub-task (aspect-specific semi

ment) we use accuracy as our target evaluation metric (Angelidis and Lapata, 2018; Sun et al., 2019). For machine translation we use Sacre-BLEU (Post, 2018).

Citation

@misc{khashabi2020parsinlu,
  title={ParsiNLU: A Suite of Language Understanding Challenges for Persian},
  author={Khashabi et al. (2020)},
  year={2020},
  note={arXiv:2012.06154}
}

• arXiv: 2012.06154