Re Mi Eval

re-mi-eval

ReMI: A Dataset for Reasoning with Multiple Images — Kazemi et al. (2024) (arXiv:2406.09175, 2024)

What this evaluates

Evaluates multimodal LLMs' ability to reason across multiple images, including sequential and set-based consumption, interleaved text-image processing, and heterogeneous visual inputs like charts, equations, maps, and code. It probes cross-image contextual integration, precise visual reading, and step-by-step logical deduction.

Datasets

• ReMI — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact match for textual outputs, ignoring spacing and case. For numeric answers, relaxed accuracy with a task-specific tolerance epsilon: default 1%, 3% for GeomShapes/GeomCost, and 10 minutes for Clocks. A prediction p is correct if (1-epsilon)*l <= p <= (1+epsilon)*l, where l is the label.
• error reduction percentage (ERP) — range: percent
  • Measures how much a model reduces error compared to a baseline B: 100 * (Error_T(B) - Error_T(M)) / Error_T(B). Normalizes the error reduction by the baseline's error magnitude.

Input / output format

Input: Multiple images (up to six) interleaved with or presented as a set/sequence of text prompts/questions. Tasks span math, physics, code, spatial/temporal reasoning, and table/chart analysis.

Output: Final answer (numeric value, text string, or multiple-choice selection). For CodeEdit, a line of code ending in a semicolon.

Scoring recipe

def compute_metric(pred, gold, task_name):
    if task_name in ['GeomShapes', 'GeomCost']:
        tol = 0.03
    elif task_name == 'Clocks':
        tol = 10  # minutes
    else:
        tol = 0.01  # 1%
    if isinstance(gold, str):
        return normalize_whitespace_case(pred) == normalize_whitespace_case(gold)
    return (1 - tol) * gold <= pred <= (1 + tol) * gold

Common pitfalls

• Relaxed accuracy tolerances are task-specific (1% default, 3% for geometry tasks, 10 minutes for clocks). Using a uniform tolerance will misreport scores.
• The naive baseline strategy varies significantly across tasks (e.g., uniform random 1/c for MC, fixed coordinates (0,0) for charts, 12*60 for clock diffs, 0 for RefCOCO).
• Models often achieve higher scores when images are provided sequentially/separately rather than concatenated into a single composite image, which can skew comparative evaluations if input format is not standardized.

Evidence (verbatim from paper)

Metrics: We mainly report accuracy for our tasks. For textual outputs, we compute exact match while handling slight variations such as spacing issues, lowercase vs uppercase, etc. For numeric answers, we compute a relaxed accuracy with 1% tolerance, mainly to avoid penalizing rounding errors. In the case of relaxed accuracy with tolerance $\epsilon$, a numeric prediction $p$ is considered correct if $(1-\epsilon)l\leq p\leq(1+\epsilon)l$ where $l$ is the label.

Citation

@misc{kazemi2024remi,
  title={ReMI: A Dataset for Reasoning with Multiple Images},
  author={Kazemi et al. (2024)},
  year={2024},
  note={arXiv:2406.09175}
}

• arXiv: 2406.09175