Kodiaqbench Eval

kodiaqbench-eval

KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark — Jang et al. (2024) (arXiv:2402.17377, 2024)

What this evaluates

Evaluates conversational understanding and response generation capabilities of language models in Korean. It probes dialogue comprehension (classifying topics, emotions, relations, dialog acts, and facts) and response selection (choosing or generating appropriate next utterances across various Korean dialogue contexts).

Datasets

• KoDialogBench — total ?; splits: test (-1); repo https://github.com/sb-jang/kodiaqlogbench

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted options or generated responses out of the total number of instances. Reported as the average accuracy across all test sets within each task suite.

Input / output format

Input: Multiple-choice prompts containing a Korean dialogue context. For comprehension tasks, a question about dialogue characteristics is appended. For response selection, a sequence of utterances ends with a speaker identifier.

Output: Model selects the correct option by calculating log-likelihoods of verbalized class names or option numbers, or generates the next utterance to complete the dialogue.

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Different sub-tasks require different prompting strategies (e.g., direct prompting vs. option prompting vs. response selection prompting); using a uniform prompt degrades performance.
• Cross-lingual instruction tuning (e.g., using English or Chinese instruction datasets) does not transfer to Korean conversational tasks and may even hurt performance.
• Class names for tasks like 'Dialog Act' are too abstract for direct prompting, requiring class descriptions to be included in the prompt.

Evidence (verbatim from paper)

We adopt the multiple-choice format, which is prevalent for evaluating language models (Hendrycks et al., 2021; Gao et al., 2023). In this approach, a language model calculates the log-likelihood of generating each option given a prompt and makes a selection accordingly. We report the accuracy results for the two task suites in Table 4 and Table 5.

Citation

@misc{jang2024kodiaqbench,
  title={KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark},
  author={Jang et al. (2024)},
  year={2024},
  note={arXiv:2402.17377}
}

• arXiv: 2402.17377