Filipino Text Benchmarks Eval

filipino-text-benchmarks-eval

Establishing Baselines for Text Classification in Low-Resource Languages — Cruz et al. (2020) (arXiv:2005.02068, 2020)

What this evaluates

Evaluates text classification capability in low-resource settings for the Filipino language, specifically measuring model robustness and performance degradation as training data size is systematically reduced.

Datasets

• Hate Speech — total ?; splits: train (-1), test (-1); repo https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
• Dengue — total ?; splits: train (-1), test (-1); repo https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: fraction of correctly predicted labels out of total test instances.
• hamming loss (primary) — range: [0, 1]
  • Fraction of incorrect labels in a multilabel setting; lower is better.
• % degradation — range: percent
  • Relative performance loss compared to the full-dataset baseline: (Baseline_Metric - Reduced_Data_Metric) / Baseline_Metric * 100.

Input / output format

Input: Raw Filipino text strings.

Output: Predicted class label(s) for each text instance.

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)

def compute_hamming_loss(preds, gold):
    total = len(preds) * len(preds[0])
    incorrect = sum(1 for p, g in zip(preds, gold) if set(p) != set(g))
    return incorrect / total

def compute_degradation(baseline_val, reduced_val):
    return ((baseline_val - reduced_val) / baseline_val) * 100

Common pitfalls

• Failing to keep the validation and test sets constant across different training data sizes, which invalidates the degradation comparison.
• Computing degradation as an absolute difference rather than the specified relative percentage.
• Not repeating the finetuning process 5 times (k=5 cross-validation) to account for variance in low-data settings.

Evidence (verbatim from paper)

Degradation is computed by subtracting the full 10k accuracy with the accuracy when trained with less samples, then dividing by the full 10k accuracy. We log the test loss and accuracy for each reduction, then compute for the differences against the loss and accuracy when trained with the full dataset. After which, we calculate the degradation percentage, which is the amount of performance loss the model experiences relative to its accuracy when trained with the full dataset.

Citation

@misc{cruz2020establishing,
  title={Establishing Baselines for Text Classification in Low-Resource Languages},
  author={Cruz et al. (2020)},
  year={2020},
  note={arXiv:2005.02068}
}

• arXiv: 2005.02068