Koffvqa Eval

koffvqa-eval

KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language — Kim et al. (2025) (arXiv:2503.23730, 2025)

What this evaluates

This benchmark evaluates the ability of large vision-language models to generate accurate, free-form Korean responses to image-based questions. It probes fine-grained capabilities across perception, reasoning, and safety/bias, specifically testing Korean cultural recognition, OCR, document/table/chart understanding, and hallucination robustness.

Datasets

• KOFFVQA — total 275; splits: test (275); repo https://github.com/maum-ai/KOFFVQA

Metrics

• KOFFVQA Score (primary) — range: [0, 100]
  • An LLM-as-a-judge evaluates each free-form response against predefined grading criteria with partial scores summing to 10. If the response language is not Korean (checked via langid), the score is set to 0, except for responses consisting entirely of numbers/special characters. The final score is the average of all question scores, multiplied by 10 to scale the range to [0, 100].

Input / output format

Input: An image paired with a Korean-language question.

Output: Unrestricted free-form text response (ideally in Korean).

Scoring recipe

def compute_koffvqa_score(predictions, grading_criteria):
    raw_scores = []
    for pred, criteria in zip(predictions, grading_criteria):
        if not langid_is_korean(pred) and not is_numeric_only(pred):
            raw_scores.append(0.0)
        else:
            raw_scores.append(llm_judge_score(pred, criteria)) # Returns 0-10
    return (sum(raw_scores) / len(raw_scores)) * 10

Common pitfalls

• Responses in languages other than Korean receive a score of 0 unless they consist purely of numbers/special characters.
• The final reported metric is scaled by 10 (range 0-100), so readers must not confuse it with the raw 0-10 judge output.
• The evaluation relies on an LLM judge guided by explicit partial-point criteria, which differs from standard exact-match or BLEU metrics used in closed-form VQA.

Evidence (verbatim from paper)

The evaluation process of KOFFVQA employs the LLM-as-a-judge approach, where the model is prompted to evaluate responses based on pre-defined grading criteria. ... The response is then fed to the LLM judge along with the predefined grading criteria, ensuring that the LLM judge is guided by these criteria to generate a score between 0 and 10 in an easily parsable format. ... After the final scores for each question are extracted, they are averaged both per subcategory and over the entire benchmark. The average scores are multiplied by 10 in order to shift the range of scores to be between 0 and 100 for more intuitive interpretation and easier comparison with other benchmarks.

Citation

@misc{kim2025koffvqa,
  title={KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language},
  author={Kim et al. (2025)},
  year={2025},
  note={arXiv:2503.23730}
}

• arXiv: 2503.23730