Cole Eval

cole-eval

COLE: a Comprehensive Benchmark for French Language Understanding Evaluation — Beauchemin et al. (2025) (arXiv:2510.05046, 2025)

What this evaluates

Evaluates French language understanding across 23 diverse tasks, including sentiment analysis, paraphrase detection, grammatical judgment, reasoning, and extractive QA. It specifically probes capabilities like morphological richness, grammatical gender, syntactic nuance, and regional language variation in a zero-shot setting.

Datasets

• COLE — total ?; splits: test (-1)

Metrics

• task-specific metrics (primary) — range: percent
  • Automatically computed per task type: exact match for classification labels, and exact match or token-level overlap for text extraction/generation tasks.

Input / output format

Input: Natural language prompt describing the task, followed by the input instance (e.g., sentence, question, or context). Evaluated in a zero-shot setting.

Output: Either a selection from a predefined set of labels (for classification/NLI) or a generated text answer (for extraction/QA).

Scoring recipe

def score(prediction, gold, task_type):
    if task_type == 'classification':
        return 1.0 if prediction == gold else 0.0
    elif task_type == 'extraction':
        return 1.0 if prediction.strip() == gold.strip() else 0.0
    return 0.0

total = sum(score(p, g, t) for p, g, t in zip(predictions, golds, task_types))
return total / len(predictions)

Common pitfalls

• Zero-shot setup means no task-specific fine-tuning or prompt engineering beyond the base NL description; results reflect pretrained capabilities only.
• Random baseline uses seed 42 and whitespace-splitting for extraction tasks, which may not reflect realistic model behavior but serves as a strict lower bound.
• Shared test set across all 95 models ensures fair comparison but may introduce dataset bias if models were exposed during pretraining.

Evidence (verbatim from paper)

Depending on the nature of the task, models either select a response from a predefined set of labels or generate an answer. Evaluation is conducted automatically using task-specific metrics.

Citation

@misc{beauchemin2025cole,
  title={COLE: a Comprehensive Benchmark for French Language Understanding Evaluation},
  author={Beauchemin et al. (2025)},
  year={2025},
  note={arXiv:2510.05046}
}

• arXiv: 2510.05046