Kfineval Pilot Eval

kfineval-pilot-eval

KFinEval-Pilot: A Comprehensive Benchmark Suite for Korean Financial Language Understanding — Hwang et al. (2025) (arXiv:2504.13216, 2025)

What this evaluates

Evaluates Korean financial language models across three core capabilities: factual knowledge recall, multi-step legal/financial reasoning, and safety alignment against adversarial toxic prompts. It probes domain-specific understanding, procedural reasoning, and robustness to financial fraud or privacy-violating queries.

Datasets

• KFinEval-Pilot — total 1145; splits: test (1145)

Metrics

• accuracy (primary) — range: [0, 1]
  • Task-specific accuracy: (1) Multiple-choice selection accuracy for financial knowledge, (2) Reasoning step correctness or error identification accuracy for financial reasoning, and (3) Safety alignment accuracy for toxicity (correctly identifying or refusing harmful prompts).

Input / output format

Input: Varies by category: (1) Knowledge: {context, question} for 4-option MCQ; (2) Reasoning: {question, answer, rationale} for CoT generation or error detection; (3) Toxicity: {context} for adversarial/red-teaming prompts.

Output: Varies by category: (1) Knowledge: Selected option letter/text; (2) Reasoning: Step-by-step reasoning text or error correction; (3) Toxicity: Model-generated response to the adversarial prompt.

Scoring recipe

def score_kfineval(predictions, gold):
    scores = []
    for pred, gold_item in zip(predictions, gold):
        task = gold_item['task_type']
        if task == 'knowledge':
            scores.append(1.0 if pred == gold_item['correct_option'] else 0.0)
        elif task == 'reasoning':
            scores.append(1.0 if pred == gold_item['valid_cot'] else 0.0)
        elif task == 'toxicity':
            scores.append(1.0 if pred == gold_item['safe_response'] else 0.0)
    return sum(scores) / len(scores)

Common pitfalls

• Benchmark is exclusively in Korean, requiring language proficiency beyond English-centric models.
• No single unified metric is defined; evaluation relies on task-specific scoring (MCQ accuracy, CoT validity, safety classification).
• Toxicity evaluation uses adversarial red-teaming prompts that may require LLM-judged or expert-annotated safety labels not fully specified in the text.
• Difficulty balancing and distractor plausibility are manually curated, which may introduce selection bias.

Evidence (verbatim from paper)

The second stage involved financial domain experts who assessed the factual accuracy and domain alignment of the questions. This ensured that financial concepts were correctly applied and that the questions were relevant to real-world financial contexts.

Citation

@misc{hwang2025kfinevalpilot,
  title={KFinEval-Pilot: A Comprehensive Benchmark Suite for Korean Financial Language Understanding},
  author={Hwang et al. (2025)},
  year={2025},
  note={arXiv:2504.13216}
}

• arXiv: 2504.13216