Polish Nlp Eval

polish-nlp-eval

Evaluation of Transfer Learning for Polish with a Text-to-Text Model — Chrabrowa et al. (2022) (arXiv:2205.08808, 2022)

What this evaluates

Evaluates Polish language understanding, summarization, and question answering capabilities of text-to-text models. It probes how well encoder-decoder and decoder-only architectures generalize from multilingual pre-training to monolingual Polish tasks using exact-match generation and ROUGE-based metrics.

Datasets

• KLEJ benchmark — total ?; splits: train (-1), val (-1), test (-1)
• Allegro Articles — total 33000; splits: train (-1), test (-1)
• Polish Summaries Corpus — total 569; splits: train (-1), test (-1)

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • Percentage of test instances where the generated token sequence exactly matches the gold label. For specific KLEJ tasks, F1, Spearman correlation, or MAE-based scores are used instead.
• ROUGE AVG — range: [0, 1]
  • Arithmetic mean of the F-measure scores for ROUGE-1, ROUGE-2, and ROUGE-L. Calculated as (ROUGE-1 + ROUGE-2 + ROUGE-L) / 3.

Input / output format

Input: Text-to-text prompts with descriptive prefixes (e.g., '<Prefix 1>: text 1 <Prefix 2>: text 2' for KLEJ), trimmed source texts (typically ≤1024 tokens for summarization), and question-context pairs for QA.

Output: Greedy token generation until EOS or max target length. For KLEJ, semantically significant text tokens matching the gold label. For summarization/QA, generated text sequences.

Scoring recipe

def score(predictions, golds, task_type):
    if task_type == 'classification':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif task_type == 'summarization':
        r1 = rouge_fmeasure(predictions, golds, n=1)
        r2 = rouge_fmeasure(predictions, golds, n=2)
        rl = rouge_fmeasure(predictions, golds, n='l')
        return (r1 + r2 + rl) / 3

Common pitfalls

• Input truncation to ~1024 tokens severely limits context for summarization, meaning models never see 100% of source text during training.
• Exact-match scoring is strict; only precise token matches count as correct, ignoring semantic equivalence.
• ROUGE metrics are noted as imperfect for abstractive summaries, with human upper bounds sometimes matching model performance.

Evidence (verbatim from paper)

Targets were generated over the whole vocabulary, and only an exact match was treated as the correct answer. Results are shown in Table 4 which contains arithmetic mean of (f-measure) ROUGE-1, ROUGE-2 and ROUGE-L (Lin, 2004) for each model and task.

Citation

@misc{chrabrowa2022evaluation,
  title={Evaluation of Transfer Learning for Polish with a Text-to-Text Model},
  author={Chrabrowa et al. (2022)},
  year={2022},
  note={arXiv:2205.08808}
}

• arXiv: 2205.08808