Demon Eval

demon-eval

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions — Li et al. (2023) (arXiv:2308.04152, 2023)

What this evaluates

Evaluates a model's ability to comprehend and follow complex, interleaved multimodal instructions that require inferring missing visual details and reasoning across multiple images and text turns. It probes reasoning-aware detail comprehension, image-text alignment, and sensitivity to visual context order.

Datasets

• DEMON — total ?; splits: test (-1); repo https://github.com/DCDmllm/Cheetah
• MME — total ?; splits: test (-1)
• OwlEval — total 82; splits: test (82)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly selected options in multiple-choice tasks.
• ROUGE-L — range: percent
  • Recall-oriented F1 score for the longest common subsequence between predicted and reference text, averaged over open-ended generation tasks.

Input / output format

Input: Interleaved images and text instructions, often requiring models to infer missing visual details, answer multiple-choice questions, or generate open-ended responses based on multimodal context.

Output: Model generates either a selected option (for multiple-choice) or a text response (for open-ended generation).

Scoring recipe

def compute_metrics(predictions, golds, task_types):
    acc_scores = []
    rouge_scores = []
    for pred, gold, t in zip(predictions, golds, task_types):
        if t == 'multiple_choice':
            acc_scores.append(1.0 if pred == gold else 0.0)
        elif t == 'open_ended':
            rouge_scores.append(rouge_l_f1(pred, gold))
    return {
        'accuracy': sum(acc_scores) / len(acc_scores) if acc_scores else 0,
        'rouge_l': sum(rouge_scores) / len(rouge_scores) if rouge_scores else 0
    }

Common pitfalls

• Models often describe image contents instead of answering the posed question due to weak image-text alignment training.
• Models are insensitive to the order of interleaved images, which significantly shifts instruction semantics.
• ROUGE-L may not capture reasoning quality or factual correctness for open-ended tasks.

Evidence (verbatim from paper)

we report the averaged accuracy for multiple-choice tasks and averaged ROUGE-L for open-ended generation tasks in Figure 6.

Citation

@misc{li2023fine,
  title={Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2308.04152}
}

• arXiv: 2308.04152