Worldqa Eval

worldqa-eval

WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoning — Zhang et al. (2024) (arXiv:2405.03272, 2024)

What this evaluates

Evaluates multimodal video understanding and long-chain reasoning by requiring models to integrate visual, auditory, and external world knowledge to answer open-ended and multiple-choice questions.

Datasets

• WorldQA — total ?; splits: test (-1)

Metrics

• GPT-4 open-ended score (primary) — range: [0, 100]
  • Rubric-based scoring using GPT-4: Correct (1), Incorrect (0), Partially correct (0.3), Incomplete but correct (0.5), Redundant (0.5). Scores are averaged across instances.
• CircularEval accuracy — range: [0, 1]
  • Model answers N times with circularly shifted options. If response matches an option, it counts. If not, ChatGPT evaluates semantic similarity to pick the best match. Accuracy is the fraction of correctly matched answers.

Input / output format

Input: Video frames (typically 8 uniformly sampled), audio clips, and a question (plus multiple-choice options for MCQA). Some settings use only the question or human-annotated video descriptions.

Output: Free-text answer for open-ended QA; selected option letter (A, B, C, D) for multi-choice QA.

Scoring recipe

def score_open_ended(pred, gold):
    if pred == gold: return 1.0
    if not set(pred) & set(gold): return 0.0
    if set(pred) < set(gold): return 0.5
    if set(gold) < set(pred): return 0.5
    return 0.3
def score_multichoice(preds, options, gold_idx):
    correct = 0
    for i in range(len(options)):
        shifted = options[i:] + options[:i]
        if preds[i] == shifted[gold_idx]: correct += 1
    return correct / len(options)

Common pitfalls

• Option order bias in multi-choice QA can artificially inflate accuracy without CircularEval.
• GPT-4 scoring for open-ended QA may not perfectly align with human preferences, though the paper shows correlation.
• Current LMMs often degrade in performance as the number of input frames increases, contrary to human trends.
• Audio modality is frequently underutilized or poorly processed by standard audio-text models, limiting overall performance.

Evidence (verbatim from paper)

Our scoring system for model answer $A$ against ground truth $G$ is: (1) $A=G$: Correct (1 point), (2) $A\cap G=\emptyset$: Incorrect (0 points), (3) $\emptyset<A\cap G<A\cup G$: Partially correct (0.3 points), (4) $A\subset G$, $A\neq G$: Incomplete but correct (0.5 points), (5) $G\subset A$, $A\neq G$: Redundant (0.5 points). ... We follow the approach of MMBench and use its proposed CircularEval evaluation method to evaluate model performance. CircularEval requires the model to answer each question N times, where N is the number of choices. Each iteration involves a circular shift of the options...

Citation

@misc{zhang2024worldqa,
  title={WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoning},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2405.03272}
}

• arXiv: 2405.03272