Text Classification Eval

text-classification-eval

Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling — Peng Zhou et al. (2016) (arXiv:1611.06639, 2016)

What this evaluates

Evaluates text classification performance across multiple sentiment, subjectivity, question classification, and topic categorization tasks. It probes the model's ability to capture contextual and syntactic features from sequential text using 2D matrix representations and spatial pooling.

Datasets

• MR — total ?; splits: train (-1), test (-1)
• SST-1 — total ?; splits: train (-1), test (-1)
• SST-2 — total ?; splits: train (-1), test (-1)
• Subj — total ?; splits: train (-1), test (-1)
• TREC — total ?; splits: train (-1), test (-1)
• 20Newsgroups — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Correct predictions divided by total predictions across all instances.
• Macro-F1 — range: [0, 1]
  • Unweighted mean of the F1 score computed independently for each class, then averaged across all classes.

Input / output format

Input: Raw text sentences or phrases, tokenized and mapped to a 2D matrix via 300-dimensional pre-trained word embeddings and a bidirectional LSTM layer.

Output: Discrete class labels matching the dataset's taxonomy (e.g., binary sentiment, 5-class sentiment, subjectivity, 6-class question type, or 4-class topic).

Scoring recipe

def compute_metric(predictions, gold_labels, metric_type):
    if metric_type == 'accuracy':
        return sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    elif metric_type == 'Macro-F1':
        classes = sorted(set(gold_labels))
        f1_scores = []
        for c in classes:
            tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
            fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
            fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
            f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
            f1_scores.append(f1)
        return sum(f1_scores) / len(f1_scores)

Common pitfalls

• SST-2 training set is significantly larger than listed in summary tables because phrases are used for training but only sentences are scored at test time.
• 20Newsgroups uses a specific 4-category subset (comp, politics, rec, religion) and requires Macro-F1 instead of accuracy.
• Hyperparameters were tuned exclusively on the SST-1 development set, so direct application to other datasets without re-tuning may yield suboptimal results.

Evidence (verbatim from paper)

The evaluation metric of the 20Ng is the Macro-F1 measure followed by the state-of-the-art work and the other five datasets use accuracy as the metric.

Citation

@misc{zhou2016textclassification,
  title={Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling},
  author={Peng Zhou et al. (2016)},
  year={2016},
  note={arXiv:1611.06639}
}

• arXiv: 1611.06639