Seacrowd Benchmark Eval

seacrowd-benchmark-eval

SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages — Holy Lovenia et al. (arXiv:2406.10118, 2024)

What this evaluates

Evaluates the zero-shot capability of LLMs, VLMs, and speech models across 13 NLU/NLG tasks, ASR, and image captioning for Southeast Asian languages. It probes multilingual understanding, generation, and cross-modal alignment in low-resource and indigenous language settings.

Datasets

• SEACrowd NLU — total ?; splits: test (-1); repo https://github.com/SEACrowd/seacrowd-experiments
• SEACrowd NLG — total ?; splits: test (-1); repo https://github.com/SEACrowd/seacrowd-experiments
• SEACrowd ASR — total ?; splits: test (-1); repo https://github.com/SEACrowd/seacrowd-experiments
• SEACrowd VL — total ?; splits: test (-1); repo https://github.com/SEACrowd/seacrowd-experiments

Metrics

• weighted F1 score (primary) — range: [0, 1]
  • Macro/micro-weighted F1 computed over class predictions for NLU tasks. Accounts for class imbalance by weighting each class's contribution by its support.
• chrF++ — range: [0, 1]
  • n-gram character n-gram F-score with word n-grams added. Measures character-level overlap between reference and hypothesis, robust to tokenization differences.
• ROUGE-L — range: [0, 1]
  • Recall and precision based on the longest common subsequence (LCS) between reference and generated text. F1 is typically reported.
• CIDEr — range: [0, 1]
  • TF-IDF weighted n-gram similarity between reference captions and hypothesis. Emphasizes rare, informative words in image captioning evaluation.
• WER (primary) — range: [0, 1] or percent
  • Word Error Rate: (insertions + deletions + substitutions) / total reference words. Used for ASR on Latin-script languages.
• CER — range: [0, 1] or percent
  • Character Error Rate: (insertions + deletions + substitutions) / total reference characters. Used for ASR on non-Latin script languages.

Input / output format

Input: Zero-shot prompts in English (3 templates for NLU, 1 for NLG) or English + target SEA language (1 each for VL). Inputs include text, images, or audio corresponding to the specific task and language subset.

Output: Model-generated predictions: class labels or scores for NLU, generated text for NLG, image captions for VL, and transcriptions for ASR.

Scoring recipe

def compute_metrics(predictions, references, task, script_type):
    if task == 'NLU':
        return f1_score(references, predictions, average='weighted')
    elif task in ['NLG', 'VL']:
        return {'chrF++': chrf(references, predictions), 'ROUGE-L': rouge_l(references, predictions)}
    elif task == 'ASR':
        metric = 'WER' if script_type == 'latin' else 'CER'
        return error_rate(references, predictions, metric=metric)

Common pitfalls

• Zero-shot evaluation may not reflect fine-tuned performance on low-resource languages, potentially underestimating model capabilities.
• ASR metrics switch between WER and CER based on script, complicating direct cross-lingual comparison without normalization.
• VL evaluation is limited to only 4 languages due to scarce human-annotated multimodal data, limiting generalizability.
• Prompt templates are predominantly in English, potentially biasing multilingual model performance and favoring English-centric architectures.

Evidence (verbatim from paper)

We conduct all evaluations in a zero-shot fashion. We employ 3 prompt templates in English for each NLU task and 1 for each NLG task. We utilize the weighted F1 score to measure the model performance on NLU tasks and n-gram reference-based metrics, i.e., chrF++ and ROUGE-L, on NLG tasks. As for VL, aside from a prompt template in English, we also use a prompt template in the respective SEA indigenous language per data subset. We report CIDEr for the image captioning task. For ASR, we use word error rate (WER) for languages with Latin script and character error rate (CER) for those with non-Latin script.

Citation

@misc{lovenia2024seacrowd,
  title={SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages},
  author={Holy Lovenia et al.},
  year={2024},
  note={arXiv:2406.10118}
}

• arXiv: 2406.10118