Vlue Eval

vlue-eval

VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding — Do et al. (2024) (arXiv:2403.15882, 2024)

What this evaluates

Evaluates Vietnamese natural language understanding across five diverse tasks including machine reading comprehension, natural language inference, emotion recognition, hate speech detection, and part-of-speech tagging. It assesses a model's ability to comprehend text, reason over sentence pairs, classify emotions and hate speech, and perform syntactic analysis in Vietnamese.

Datasets

• UIT-ViQuAD 2.0 — total 35990; splits: train (28457), dev (3821), test (3712)
• ViNLI — total 30376; splits: train (24376), dev (3009), test (2991)
• VSMEC — total 6927; splits: train (5548), dev (686), test (693)
• ViHOS — total 11214; splits: train (8974), dev (1112), test (1128)
• NIIVTB POS — total 20588; splits: train (18588), dev (1000), test (1000)

Metrics

• Exact Match (EM) — range: [0, 1]
  • 1 if the predicted answer exactly matches the gold answer span, else 0.
• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall computed over token or character overlap between predicted and gold spans/labels.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.
• Macro-F1 — range: [0, 1]
  • Unweighted mean of F1 scores computed independently for each class, then averaged.

Input / output format

Input: Varies by task: (context, question) for MRC; (premise, hypothesis) for NLI; (comment text) for emotion/hate speech; (sentence tokens) for POS tagging.

Output: Varies by task: extracted span or empty string for MRC; class label (entailment/neutral/contradiction/other) for NLI; emotion label(s) for emotion recognition; hate/offensive span(s) or none for hate speech; POS tag sequence for POS tagging.

Scoring recipe

def compute_metrics(preds, golds):
    em = sum(1.0 if p == g else 0.0 for p, g in zip(preds, golds)) / len(golds)
    f1s = []
    for p, g in zip(preds, golds):
        p_set, g_set = set(p.split()), set(g.split())
        if not p_set and not g_set: f1s.append(1.0)
        elif not p_set or not g_set: f1s.append(0.0)
        else:
            prec = len(p_set & g_set) / len(p_set)
            rec = len(p_set & g_set) / len(g_set)
            f1s.append(2 * prec * rec / (prec + rec) if prec + rec > 0 else 0.0)
    return {'EM': em, 'F1': sum(f1s) / len(f1s)}

Common pitfalls

• Models must predict an empty span for unanswerable questions in UIT-ViQuAD; failing to do so penalizes EM/F1.
• VSMEC is a multi-label classification task, so macro-F1 must be computed per label rather than using standard single-label accuracy.
• ViHOS requires span-level extraction, not just comment-level classification, so evaluation must match predicted spans to gold spans character-by-character.

Evidence (verbatim from paper)

The task proposed by this dataset is to extract the answer for a question given a corresponding context. The answer can be empty when models encounter unanswerable questions. Exact Match (EM) and F1-score are used to evaluate the performance of the model.

Citation

@misc{do2024vlue,
  title={VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding},
  author={Do et al. (2024)},
  year={2024},
  note={arXiv:2403.15882}
}

• arXiv: 2403.15882