Jcola Eval

jcola-eval

JCoLA: Japanese Corpus of Linguistic Acceptability — Someya et al. (2023) (arXiv:2309.12676, 2023)

What this evaluates

This benchmark evaluates the ability of neural language models to judge the grammatical acceptability of Japanese sentences. It probes deep syntactic knowledge, particularly long-distance dependencies and linguistic phenomena, by measuring performance on both in-domain and out-of-domain acceptability judgments.

Datasets

• JCoLA — total 10020; splits: dev (-1); repo https://github.com/osekilab/JCoLA

Metrics

• Matthews Correlation Coefficient (MCC) (primary) — range: [-1, 1]
  • A correlation coefficient between the observed and predicted binary classifications. It accounts for true and false positives and negatives, making it robust for unbalanced binary classification tasks.

Input / output format

Input: A single Japanese sentence.

Output: Binary acceptability label (acceptable vs. unacceptable).

Scoring recipe

def evaluate_jcola(model, dev_data, test_data):
    best_avg_test_mcc = -1.0
    for lr in [5e-5, 3e-5, 2e-5]:
        dev_mccs = []
        test_mccs = []
        for seed in range(20):
            m = train_and_finetune(model, dev_data, lr, seed, epochs=5)
            dev_mcc = compute_mcc(m.predict(dev_data), dev_data.labels)
            if dev_mcc < 0:
                continue
            dev_mccs.append(dev_mcc)
            test_mccs.append(compute_mcc(m.predict(test_data), test_data.labels))
        if test_mccs:
            avg_test_mcc = sum(test_mccs) / len(test_mccs)
            if avg_test_mcc > best_avg_test_mcc:
                best_avg_test_mcc = avg_test_mcc
    return best_avg_test_mcc

Common pitfalls

• The dataset is unbalanced, so accuracy is misleading; MCC must be used as the primary metric.
• Any configuration or seed yielding a negative MCC on the development set must be discarded entirely.
• Final scores must be averaged across 20 different random seeds per configuration to mitigate randomness.

Evidence (verbatim from paper)

In addition, the language models are trained using three different learning rates (5e-5, 3e-5, and 2e-5) and we evaluate models which achieved the highest Matthews Correlation Coefficient (MCC; Matthews (1975)) on the development data. This evaluation metric is an evaluation metric suitable for unbalanced binary classifiers also used in Warstadt et al. (2019). For each configuration, we trained 20 models with different random seeds to mitigate the effect of randomness. The score for each language model is calculated as the average across 20 different random seeds, but we ignore those results where the models achieved less than zero MCC score on the development set, as in Warstadt and Bowman (2020).

Citation

@misc{someya2023jcola,
  title={JCoLA: Japanese Corpus of Linguistic Acceptability},
  author={Someya et al. (2023)},
  year={2023},
  note={arXiv:2309.12676}
}

• arXiv: 2309.12676