Cello Eval

cello-eval

CELLO: Causal Evaluation of Large Vision-Language Models — Chen et al. (2024) (arXiv:2406.19131, 2024)

What this evaluates

This benchmark evaluates the causal reasoning capabilities of Large Vision-Language Models (LVLMs) across four levels of the Ladder of Causation: discovery, association, intervention, and counterfactual. It probes whether models can correctly identify causal relationships, handle confounding and collider biases, reason about interventions, and generate counterfactual explanations based on visual scenes.

Datasets

• CELLO — total 14094; splits: test (1200); repo https://github.com/OpenCausaLab/CELLO

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard accuracy calculated as the proportion of correctly predicted answers out of the total number of test instances. Random baselines are set at 0.5 for binary questions and 0.25 for multiple-choice questions.

Input / output format

Input: An image depicting a scene with human, object, or human-human interactions, accompanied by a multiple-choice or binary question probing causal relationships (e.g., causality identification, confounder identification, natural direct effect).

Output: A single selected option from the provided choices (e.g., A, B, C, D) or a binary Yes/No response. The model may also output chain-of-thought reasoning if prompted with CELLO-CoT.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p.strip().upper() == g.strip().upper())
    return correct / len(gold_labels)

Common pitfalls

• Models often fail to format outputs correctly, leading to 'Unformatted Answer' or 'Uncertain Answer' errors that are hard to parse.
• During robustness testing, models tend to prioritize the polite tone of reformulated prompts over the actual causal logic, causing accuracy to plummet (e.g., from ~0.5 to ~0.03).
• Random baselines differ by question type (0.5 for binary, 0.25 for MCQ), so comparing raw scores without accounting for question distribution can be misleading.

Evidence (verbatim from paper)

We compose a test set consisting of 1,200 samples, distributed equally across 12 causal tasks in CELLO, with each task featuring 100 randomly selected samples. For consistent evaluation, we use standard accuracy metrics for all the models and tasks. Performance is also benchmarked against a random baseline (i.e., 0.5 for binary and 0.25 for multiple-choice questions).

Citation

@misc{chen2024cello,
  title={CELLO: Causal Evaluation of Large Vision-Language Models},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2406.19131}
}

• arXiv: 2406.19131