qhjqhj00 dutch-book-review-sentiment-eval
The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews — van der Burgh et al. (2019) (arXiv:1910.00896, 2019)
What this evaluates
Evaluates the effectiveness of Universal Language Model Fine-tuning (ULMFiT) versus traditional SVM classifiers on low-resource, domain-specific sentiment classification tasks using small training sets.
Datasets
- Dutch book reviews — total 15000; splits: train (15000)
Metrics
accuracy(primary) — range: percent- The proportion of correctly classified book reviews out of the total number of reviews in the evaluation set, expressed as a percentage.
Input / output format
Input: Raw Dutch book review text, tokenized by whitespace and padded to uniform length within batches.
Output: Binary sentiment label (positive or negative).
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Hyperparameters were optimized separately for each training set size and fold, which may not generalize to other dataset sizes.
- TF-IDF features for the SVM baseline were trained on the same 105k documents used for ULMFiT pre-training, potentially inflating baseline performance compared to a strictly in-domain TF-IDF setup.
- Small dataset sizes (100–1600 samples) make results highly sensitive to random seed and fold partitioning.
Evidence (verbatim from paper)
For comparison we also trained two models, one SVM and one ULMFiT model, with manually tuned hyperparameters on all available book reviews in the training set (15k). These models achieved 93.84% (ULMFiT) and 89.16% (SVM).
Citation
@misc{vanderburgh2019ulmfit,
title={The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews},
author={van der Burgh et al. (2019)},
year={2019},
note={arXiv:1910.00896}
}
- arXiv: 1910.00896