Germeval17 Eval

germeval17-eval

Re-Evaluating GermEval17 Using German Pre-Trained Language Models — Aßenmacher et al. (2021) (arXiv:2102.12330, 2021)

What this evaluates

Evaluates German NLP models on aspect-based sentiment analysis (ABSA) tasks, including relevance classification, document-level polarity, aspect/sentiment classification, and opinion target extraction.

Datasets

• GermEval17 — total ?; splits: test_syn (-1), test_dia (-1); repo https://github.com/ac74/reevaluating-germeval2017

Metrics

• micro F1 (primary) — range: [0, 1]
  • Micro-averaged F1 score computed across all classes, treating each instance equally regardless of class frequency.
• entity-level micro F1 (exact match) — range: [0, 1]
  • Entity-level micro F1 score requiring exact token boundary matches between predicted and gold entities.
• entity-level micro F1 (overlapping match) — range: [0, 1]
  • Entity-level micro F1 score tolerating a deviation of +/- one token in entity boundaries.

Input / output format

Input: German social media text documents. For token-level tasks, sequence positions are provided to map tokens to BIO tags.

Output: For classification tasks: predicted class labels (e.g., true/false, positive/neutral/negative, aspect categories, sentiments). For token-level tasks: BIO-tag sequences (B-, I-, O) for each token.

Scoring recipe

def compute_micro_f1(predictions, gold):
    tp = fp = fn = 0
    for p, g in zip(predictions, gold):
        if p == g: tp += 1
        elif p != g:
            fp += 1
            fn += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Multiple mentions of the same label in one document are not counted separately for Subtask C.
• Comparing results against models that used different data splits (e.g., Guhr et al. 2020) limits comparability.
• Overlapping match (D2) tolerates a deviation of +/- one token in entity boundaries, which significantly changes scoring compared to exact match.

Evidence (verbatim from paper)

Subtask A The Relevance Classification is a binary document classification task with classes true and false. Table 7 displays the micro F1 score obtained by each language model on each test set (best result per data set in bold).

Citation

@misc{assenmacher2021reevaluatinggermeval17,
  title={Re-Evaluating GermEval17 Using German Pre-Trained Language Models},
  author={Aßenmacher et al. (2021)},
  year={2021},
  note={arXiv:2102.12330}
}

• arXiv: 2102.12330