Korean Vlm Benchmarks Eval

korean-vlm-benchmarks-eval

VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models — Ju et al. (2024) (arXiv:2411.19103, 2024)

What this evaluates

Evaluates vision-language models on Korean multimodal comprehension, document/table/chart understanding, and open-ended generation capabilities using translated and newly curated benchmarks.

Datasets

• K-MMBench — total ?; splits: dev (-1); HF NCSOFT/K-MMBench
• K-SEED — total ?; splits: test (-1); HF NCSOFT/K-SEED
• K-MMStar — total 1500; splits: test (1500); HF NCSOFT/K-MMStar
• K-DTCBench — total 240; splits: test (240); HF NCSOFT/K-DTCBench
• K-LLaVA-W — total 60; splits: test (60); HF NCSOFT/K-LLaVA-W

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total questions) * 100.
• JudgeLLM ratio score — range: percent
  • Ratio of the target model's JudgeLLM score to the baseline model's JudgeLLM score. JudgeLLM (GPT-4o) evaluates helpfulness, relevance, accuracy, level of detail, and Korean generation capability based on provided captions.

Input / output format

Input: An image paired with a Korean question (multiple-choice or open-ended).

Output: For MCQA: a single choice letter or answer string. For K-LLaVA-W: a free-form Korean text response.

Scoring recipe

# For MCQA datasets (K-MMBench, K-SEED, K-MMStar, K-DTCBench)
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
accuracy = (correct / len(golds)) * 100

# For K-LLaVA-W
target_scores = [judge_llm.evaluate(q, img, pred, caption) for q, img, pred, caption in zip(questions, images, predictions, captions)]
baseline_scores = [judge_llm.evaluate(q, img, baseline_pred, caption) for q, img, baseline_pred, caption in zip(questions, images, baselines, captions)]
ratio_score = (sum(target_scores) / sum(baseline_scores)) * 100

Common pitfalls

• Translation artifacts and localization issues require human post-editing to ensure fluency and accuracy.
• K-MMStar originally contained unanswerable cases (e.g., requiring multiple images); questions were modified or recreated to ensure single-image answerability.
• K-LLaVA-W evaluation relies on GPT-4o as both JudgeLLM and baseline generator, which may introduce model-specific biases compared to standard LLM-as-a-judge setups.

Evidence (verbatim from paper)

JudgeLLM measures the model’s helpfulness, relevance, accuracy, level of detail, and Korean generation capability. The final K-LLaVA-W score of the target model is calculated as the ratio of the target model’s JudgeLLM score to the baseline model’s JudgeLLM score.

Citation

@misc{ju2024varcovision,
  title={VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models},
  author={Ju et al. (2024)},
  year={2024},
  note={arXiv:2411.19103}
}

• arXiv: 2411.19103