Filbench Eval

filbench-eval

FilBench: Can LLMs Understand and Generate Filipino? — Miranda et al. (2025) (arXiv:2508.03523, 2025)

What this evaluates

Evaluates LLMs' ability to understand and generate text in Filipino, Tagalog, and Cebuano across cultural knowledge, classical NLP tasks, reading comprehension, and text generation. It probes cultural alignment, factual recall, linguistic processing, and translation capabilities in low-resource Southeast Asian languages.

Datasets

• FilBench — total ?; splits: test (-1); repo https://github.com/filbench/filbench-eval

Metrics

• accuracy — range: [0, 1]
  • Number of correct answers divided by the total number of examples. Applied to Cultural Knowledge, Classical NLP, and Reading Comprehension categories.
• ROUGE-L — range: [0, 1]
  • ROUGE-L score computed between the LLM-generated text and the gold reference text. Applied to the Generation category.
• FilBench Score (primary) — range: [0, 100]
  • Weighted average of per-category scores based on the number of examples: 100 * sum(n_i * S_i) / sum(n_i), where n_i is the example count and S_i is the category score.

Input / output format

Input: Multiple-choice or open-ended prompts for CK, CN, and RC tasks following the MCF formulation; text prompts for GN tasks (translation, summarization, etc.).

Output: For CK/CN/RC: selected answer option or generated text. For GN: generated text string.

Scoring recipe

def compute_metrics(predictions, golds, category):
    if category in ['CK', 'CN', 'RC']:
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds)
    elif category == 'GN':
        return compute_rouge_l(predictions, golds)

def compute_filbench_score(category_scores, category_sizes):
    total_examples = sum(category_sizes)
    weighted_sum = sum(n * s for n, s in zip(category_sizes, category_scores))
    return 100 * weighted_sum / total_examples

Common pitfalls

• The overall FilBench Score is a weighted average by example count across categories, not a simple arithmetic mean.
• Generation tasks are evaluated using ROUGE-L, which may not fully capture translation quality or instruction-following failures compared to human evaluation.
• Datasets span multiple languages (Filipino, Tagalog, Cebuano) and task types, so results should not be averaged without considering category weights and language distribution.

Evidence (verbatim from paper)

The CN, CK, and RC categories follow the MCF task formulation, so we score an LLM’s performance for these categories by computing the accuracy, i.e., the number of correct answers divided by the total number of examples. For GN, we compute the ROUGE-L score between the LLM-generated text and the gold reference text. All per-category metrics range from 0 to 1. In order to create a representative, single evaluation score, we perform a weighted average based on the number of examples across results as shown in [Equation 1]: FilBench Score = 100 * sum(n_i * S_i) / sum(n_i)

Citation

@misc{miranda2025filbench,
  title={FilBench: Can LLMs Understand and Generate Filipino?},
  author={Miranda et al. (2025)},
  year={2025},
  note={arXiv:2508.03523}
}

• arXiv: 2508.03523