Kalahi Eval

kalahi-eval

Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino — Montalan et al. (2024) (arXiv:2409.15380, 2024)

What this evaluates

This benchmark probes an LLM's ability to understand and generate culturally appropriate responses for Filipino contexts. It evaluates whether models can align with the lived experiences, values, and preferred strategies of action of average native Filipino speakers across nuanced socio-cultural scenarios.

Datasets

• Kalahi — total 150; splits: test (150); repo https://github.com/aisingapore/kalahi

Metrics

• MC1 (primary) — range: [0, 1]
  • The score is 1 if the model assigns the highest byte-length-normalized log-probability of completion to the best response among the choices, otherwise 0.
• MC2 — range: [0, 1]
  • The score is the sum of byte-length-normalized log-probabilities of all relevant responses, divided by the sum of normalized log-probabilities of all relevant and irrelevant responses.
• Open-ended similarity difference — range: [-1, 1]
  • The score is the maximum similarity between the generated completion and any relevant reference response, minus the maximum similarity between the completion and any irrelevant reference response. Similarity is computed using BLEURT, BERTScore, ChrF++, ROUGE-L, or METEOR.

Input / output format

Input: A culturally specific scenario prompt in Filipino. For multiple-choice tasks, the prompt is followed by a set of candidate responses (one best, others irrelevant). For open-ended tasks, only the scenario prompt is provided.

Output: For MC tasks: the model's completion log-probabilities for each candidate response. For open-ended: a single natural language response generated via greedy decoding with a maximum of 256 tokens.

Scoring recipe

def score_mc1(best_resp, irr_resps, model):
    best_logp = model.log_prob(best_resp) / len(best_resp)
    irr_logps = [model.log_prob(r) / len(r) for r in irr_resps]
    return 1.0 if best_logp > max(irr_logps) else 0.0

def score_mc2(rel_resps, all_resps, model):
    rel_sum = sum(model.log_prob(r) / len(r) for r in rel_resps)
    all_sum = sum(model.log_prob(r) / len(r) for r in all_resps)
    return rel_sum / all_sum

def score_openended(gen, rel_resps, irr_resps, sim_fn):
    max_rel = max(sim_fn(gen, r) for r in rel_resps)
    max_irr = max(sim_fn(gen, r) for r in irr_resps)
    return max_rel - max_irr

Common pitfalls

• Models may struggle with standard MC formats due to order sensitivity or first-token probability biases; Kalahi mitigates this by scoring log-probabilities of full completions separately.
• Overlap metrics like ROUGE-L and BLEU are unreliable for Filipino due to agglutinative morphology causing low lexical overlap despite semantic correctness.
• Cultural hallucinations or inappropriate suggestions are not automatically penalized by the metrics and require manual validation.

Evidence (verbatim from paper)

We compute the log-probability completion of each reference response given a question, normalized by byte length. Two scores are calculated: MC1: Choices include the best and irrelevant responses. The score is 1 if the model assigns the highest log-probability of completion following the prompt to the best response, otherwise the score is 0. MC2: Choices include all relevant and irrelevant responses. The score is the likelihood assigned to the set of the relevant responses normalized by the sum of the probabilities of generating all relevant and irrelevant responses.

Citation

@misc{montalan2024kalahi,
  title={Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino},
  author={Montalan et al. (2024)},
  year={2024},
  note={arXiv:2409.15380}
}

• arXiv: 2409.15380