Wangchanthaiinstruct Eval

wangchanthaiinstruct-eval

WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thai — Limkonchotiwat et al. (2025) (arXiv:2508.15239, 2025)

What this evaluates

Evaluates instruction-following capabilities of LLMs in Thai across culture-aware, domain-specific (Medical, Law, Finance, Retail), and multitask settings. Probes factual accuracy, reasoning quality, and fluency in both zero-shot and fine-tuned regimes.

Datasets

• WangchanThaiInstruct — total ?; splits: train (-1), test (-1)
• Thai LLM Leaderboard — total ?; splits: test (-1)
• Thai MT-Bench — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 100]
  • LLM-as-a-judge evaluates factual correctness against gold answers for QA and classification tasks. Accuracy is the percentage of responses deemed factually correct.
• Rating (1-10) — range: [1, 10]
  • LLM-as-a-judge assigns a score from 1 to 10 based on the quality of reasoning, justification, fluency, or cohesiveness depending on the task type.

Input / output format

Input: Thai instruction prompts spanning seven task types (Brainstorming, Classification, Closed QA, Creative Writing, Multiple Choice, Open QA, Summarization) across four domains (Medical, Law, Finance, Retail).

Output: Model-generated Thai text responses corresponding to each instruction prompt.

Scoring recipe

def evaluate(predictions, gold, task_type):
    if task_type in ['QA', 'Classification', 'Multiple Choice']:
        is_correct = llm_judge_check_correctness(predictions, gold)
        accuracy = sum(is_correct) / len(predictions) * 100
        rating = llm_judge_rate_reasoning(predictions, gold)
        return {'Accuracy': accuracy, 'Rating (1-10)': rating}
    else:
        fluency = llm_judge_rate_fluency(predictions)
        rating = llm_judge_rate_quality(predictions)
        return {'Fluency (1-10)': fluency, 'Rating (1-10)': rating}

Common pitfalls

• Using BLEU or ROUGE-L as primary metrics, which the authors explicitly note fail to capture reasoning quality and valid alternative phrasings.
• Assuming out-of-domain benchmarks (Thai LLM Leaderboard) are culturally aligned; they are human-verified translations without cultural grounding, potentially skewing generalization assessments.
• Ignoring the LLM-as-a-judge prompt variations; the paper uses different prompts for objective tasks vs. creative tasks, and mixing them invalidates the 1-10 rating scale.

Evidence (verbatim from paper)

Traditional metrics such as BLEU or ROUGE-L fail to capture this dimension, since two correct explanations may differ in wording or format yet still arrive at the same valid conclusion. Building on the foundation of MTBench, we utilize LLMs to assess both correctness for tasks with objective answers (e.g., QA tasks) and to assign a 1–10 rating based on the quality of reasoning or justification. For tasks requiring definitive answers, such as question answering and classification, our evaluation is twofold: we verify the factual accuracy and evaluate the accompanying reasoning against human-labeled annotations that justify correct or incorrect responses.

Citation

@misc{limkonchotiwat2025wangchanthaiinstruct,
  title={WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thai},
  author={Limkonchotiwat et al. (2025)},
  year={2025},
  note={arXiv:2508.15239}
}

• arXiv: 2508.15239