Formosanbench Eval

formosanbench-eval

FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models — Lin et al. (2025) (arXiv:2506.21563, 2025)

What this evaluates

Evaluates large language models and speech systems on three endangered Formosan Austronesian languages (Atayal, Amis, Paiwan) across machine translation, automatic speech recognition, and text summarization. It probes zero-shot, few-shot (10-shot), and fine-tuning adaptation capabilities in typologically complex, low-resource settings.

Datasets

• FormosanBench — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: [0, 1]
  • Measures n-gram precision (typically up to 4-grams) between model outputs and reference translations, with a brevity penalty to discourage overly short outputs.
• WER — range: [0, 1]
  • Quantifies the proportion of word-level errors (insertions, deletions, substitutions) in transcribed speech relative to the reference.
• ROUGE — range: [0, 1]
  • Evaluates summary quality based on overlapping units such as bigrams (ROUGE-2) or longest common subsequences (ROUGE-L) between generated and reference summaries.

Input / output format

Input: Text prompts with 10-shot in-context examples for MT and summarization; raw audio recordings for ASR.

Output: Target language text for MT, transcribed text for ASR, or summary text for summarization.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task == 'mt':
        return compute_bleu(reference=golds, hypothesis=predictions, max_order=4)
    elif task == 'asr':
        return edit_distance(golds, predictions) / len(golds)
    elif task == 'sum':
        return compute_rouge(golds, predictions, rouge_types=['rouge2', 'rougeL'])

Common pitfalls

• MMS-1b-all requires explicit language specification and only supports Amis, as Atayal and Paiwan were excluded from its pretraining corpus.
• Fine-tuning hyperparameters differ across tasks: MT/Summarization use 20 epochs with batch size 4, while ASR uses 5000 steps with batch size 16.
• GPT-4o outputs are supplemented with human judgments for fluency and content relevance, as automatic metrics alone may not capture quality.

Evidence (verbatim from paper)

We report BLEU scores (Papineni et al., [2002]) for the machine translation (MT) task, while the ASR and summarization tasks are evaluated using word error rate (WER) and ROUGE scores (Lin, [2004]), respectively. BLEU measures n-gram precision (typically up to 4-grams) between model outputs and reference translations, with a brevity penalty to discourage overly short outputs. WER quantifies the proportion of word-level errors (insertions, deletions, substitutions) in transcribed speech, while ROUGE evaluates summary quality based on overlapping units such as bigrams (ROUGE-2) or longest common subsequences (ROUGE-L) between generated and reference summaries.

Citation

@misc{lin2025formosanbench,
  title={FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2506.21563}
}

• arXiv: 2506.21563