Czenecclassification Eval

czenecclassification-eval

A Dataset and Strong Baselines for Classification of Czech News Texts — Hynek Kydlíček, Jindřich Libovický (2023) (arXiv:2307.10666, 2023)

What this evaluates

Evaluates NLP models on four challenging text classification tasks using a large-scale Czech news dataset: identifying the news source, article category, inferred author gender, and publication day of the week. The benchmark tests a model's ability to capture deep textual dependencies and contextual cues beyond simple keyword matching.

Datasets

• CZE-NEC — total 1600000; splits: train (-1), test (-1), test_small (10000), test_human (-1); repo https://github.com/hynky1999/Czech-News-Classification-dataset

Metrics

• F1 Macro (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across all classes for each task. Calculated as the unweighted mean of the F1 scores for each individual class.

Input / output format

Input: Raw Czech news article text (truncated to 510 tokens for Transformer models).

Output: Single class label per task (e.g., news source name, category, author gender, or day of the week).

Scoring recipe

def compute_macro_f1(predictions, gold_labels, classes):
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != cls and g == cls)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Models may achieve high accuracy by simply spotting keywords for specific classes rather than understanding deeper context.
• Text truncation to 510 tokens may discard relevant information for tasks like author gender or day of week.
• Human baseline performance is low due to inherent ambiguity, making human comparison scores misleading if not contextualized with inter-annotator agreement.

Evidence (verbatim from paper)

Table 3. Tasks F1 Macro scores for selected models on the Test set. We use — to denote that the model failed to converge for all tested learning rates.

Citation

@misc{kydlicek2023czenecczechnews,
  title={A Dataset and Strong Baselines for Classification of Czech News Texts},
  author={Hynek Kydlíček, Jindřich Libovický (2023)},
  year={2023},
  note={arXiv:2307.10666}
}

• arXiv: 2307.10666