Twiff Bench Eval

twiff-bench-eval

TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning — Liu et al. (2026) (arXiv:2602.10675, 2026)

What this evaluates

Evaluates a model's ability to perform dynamic visual reasoning by generating temporally grounded, physically plausible future frames and textual explanations. It probes both the quality of the step-by-step reasoning process and the correctness of the final answer in open-ended video scenarios.

Datasets

• TwiFF-Bench — total ?; splits: test (-1); repo https://github.com/LiuJunhua02/TwiFF
• Seed-Bench-R1 — total ?; splits: test (-1)

Metrics

• CoT score — range: [0, 5]
  • LLM-judged score from 0 to 5 evaluating the plausibility, temporal alignment, and physical correctness of the generated reasoning chain (interleaved text and future frames).
• Answer score (primary) — range: [0, 5]
  • LLM-judged score from 0 to 5 evaluating the correctness and accuracy of the model's final answer to the prompt.

Input / output format

Input: Video clips or sequential frames paired with a textual question or instruction requiring dynamic reasoning.

Output: Interleaved sequence of generated future frames and textual reasoning steps, terminated by a final answer.

Scoring recipe

def score_model_output(model_output, ground_truth_video, question):
    # LLM judge (GPT-5.1) evaluates CoT and Answer separately
    cot_score = llm_judge.score(
        prompt=scoring_prompt_cot,
        context={"question": question, "video": ground_truth_video, "reasoning": model_output.reasoning_chain}
    ) # Returns float 0-5
    ans_score = llm_judge.score(
        prompt=scoring_prompt_ans,
        context={"question": question, "video": ground_truth_video, "answer": model_output.final_answer}
    ) # Returns float 0-5
    return cot_score, ans_score

Common pitfalls

• Models may enter infinite loops during visual Chain-of-Thought generation; the protocol explicitly caps tool invocations at 5 and image generations at 8, truncating any excess.
• Seed-Bench-R1 lacks reference reasoning chains, so only the answer score is reported for this benchmark, unlike TwiFF-Bench which reports both CoT and answer scores.
• CoT score measures reasoning quality/alignment with future events, not answer correctness; conflating the two leads to misinterpretation of model capabilities.

Evidence (verbatim from paper)

On TwiFF-Bench, we evaluate both the CoT score and the answer score of models. On Seed-Bench-R1, due to the absence of reference reasoning chains, we evaluate only the answer score. All scores are bounded between 0 and 5. The specific details of the scoring prompt are provided in the Section B.2.

Citation

@misc{liu2026twiff,
  title={TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2602.10675}
}

• arXiv: 2602.10675