Click Eval

click-eval

CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean — Kim et al. (2024) (arXiv:2403.06412, 2024)

What this evaluates

Evaluates language models' proficiency in Korean cultural knowledge (e.g., history, law, society, economy) and linguistic competence (e.g., grammar, functional usage) using multiple-choice questions sourced from official Korean exams and textbooks.

Datasets

• CLIcK — total 1995; splits: test (1995); repo https://github.com/rladmstn1714/CLIcK

Metrics

• accuracy (primary) — range: [0, 1]
  • Instance accuracy is computed as count(correct answers) / 3N, where N is the number of options. The model is evaluated 3 times per instance with cyclically permuted option orders, and the average score across all instances is reported.

Input / output format

Input: Multiple-choice questions with 4 or 5 options (A/B/C/D or A/B/C/D/E). Prompts are provided in two variants: Type 1 includes background context, Type 2 does not. Each question is presented 3 times with cyclically permuted option positions.

Output: A single uppercase alphabet corresponding to the selected option (e.g., 'A'). For API models, verbose responses are accepted if they clearly indicate one option per predefined acceptance criteria.

Scoring recipe

correct_count = 0
for instance in dataset:
    options = instance.options
    gold = instance.gold_answer
    for perm in cyclic_permutations(options, k=3):
        prompt = build_prompt(instance, perm, context=instance.context)
        response = model.generate(prompt, temperature=0)
        pred = extract_answer(response) # applies acceptance criteria
        if pred == gold:
            correct_count += 1
accuracy = correct_count / (3 * len(options))

Common pitfalls

• Failing to average over the 3 cyclic permutations of option orders, which can introduce position bias.
• Using strict exact-match string comparison for API-generated responses, whereas the protocol allows flexible extraction (e.g., 'the answer is A') based on Jin et al. criteria.
• Ignoring the distinction between Type 1 (with context) and Type 2 (without context) prompts, as model performance varies significantly between them.

Evidence (verbatim from paper)

Our primary evaluation metric is accuracy. We report the average accuracy over the entire dataset. As we prompt the model 3 times and adopt cyclic permutation for each instance, the total number of experiments per instance is $3N$, where $N$ denotes the “number of options”. Instance accuracy is computed as: $p_{ ext{accuracy}}=rac{ ext{count}( ext{correct answers})}{3N}$

Citation

@misc{kim2024click,
  title={CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2403.06412}
}

• arXiv: 2403.06412