Llava Cot Eval

llava-cot-eval

LLaVA-CoT: Let Vision Language Models Reason Step-by-Step — Xu et al. (2024) (arXiv:2411.10440, 2024)

What this evaluates

Evaluates the effectiveness of structured chain-of-thought prompting and test-time scaling algorithms on multimodal reasoning tasks. It probes whether enforcing a specific reasoning order (summary, caption, reasoning, conclusion) and selecting among multiple generated candidates improves answer accuracy over baseline prompting or dense supervision.

Datasets

• Unspecified multimodal reasoning benchmarks — total ?; splits: (unstated)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match or numerical comparison between the model's final answer in the tag and the ground truth label. Performance is reported as the proportion of correctly answered instances.

Input / output format

Input: An image paired with a natural language question, formatted with structured CoT prompting tags (, , , ) to guide the model's step-by-step generation.

Output: A structured response containing four distinct sections: (problem restatement), (visual description), (step-by-step calculation/logic), and (final answer).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_val in zip(predictions, gold):
        # Extract final answer from <CONCLUSION> tag
        final_answer = extract_tag(pred, 'CONCLUSION')
        if normalize(final_answer) == normalize(gold_val):
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Assuming dense GPT-generated supervision alone drives performance; ablation shows the structured CoT format itself is the key driver.
• Assuming any stage ordering works; training with shuffled stage orders yields almost no improvement, proving natural reasoning order is crucial.
• Confusing test-time scaling (inference-time candidate selection) with training-time scaling; the paper explicitly evaluates scaling parameters like N for Best-of-N and retracing iterations for SWIRES during inference.

Evidence (verbatim from paper)

The performance is significantly worse, suggesting that denser supervision from GPT-4o is not the reason for improvement. From the figure, it can be observed that test-time scaling effectively corrects the errors made by the model during generation.

Citation

@misc{xu2024llavacot,
  title={LLaVA-CoT: Let Vision Language Models Reason Step-by-Step},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2411.10440}
}

• arXiv: 2411.10440