Miroeval Eval

miroeval-eval

MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome — Fangda Ye et al. (2026) (arXiv:2603.28407, 2026)

What this evaluates

Evaluates multimodal deep research agents on both the quality of their final synthesized reports and the underlying investigative process. It measures adaptive synthesis quality, factual grounding against heterogeneous sources, and process-centric attributes like search breadth, analytical depth, and alignment between intermediate findings and the final report.

Datasets

• MiroEval — total ?; splits: test (-1); repo https://github.com/MiroMindAI/MiroEval

Metrics

• Adaptive Synthesis Quality (S_quality) (primary) — range: [0, 10]
  • A weighted sum of LLM-generated scores across fixed and dynamic evaluation dimensions. Dimensions and criteria are dynamically generated per query. Formula: S_quality = ∑{d∈D} W_d ∑{c} w_{d,c} s_{d,c}, where s_{d,c} ∈ [0,10].
• Agentic Factuality — range: percent
  • Decomposes the report into verifiable statements, retrieves evidence from web search and attachments, and assigns a label: RIGHT, WRONG, CONFLICT, or UNKNOWN. Evaluated via accuracy or distribution over these labels.
• Process-Centric Score (S_process) — range: [0, 10]
  • Combines intrinsic process quality and alignment scores: S_process = α S_intrinsic(P) + (1-α) S_align(P,R). Intrinsic covers search breadth, analytical depth, progressive refinement, critical thinking, and efficiency. Alignment covers P→R, R→P, and contradiction detection.

Input / output format

Input: Research instruction I, optional multimodal attachments A, and optionally raw process logs P.

Output: Final citation-backed research report R, and process logs P (for process evaluation).

Scoring recipe

def score_miroeval(query, attachments, report, process_log):
    # 1. Adaptive Synthesis Quality
    dims = generate_dimensions(query) # fixed + dynamic
    weights = assign_weights(dims)
    s_quality = 0
    for d in dims:
        crits = generate_criteria(d, query)
        for c in crits:
            s = llm_score(report, d, c, query) # [0,10]
            s_quality += weights[d] * weights[c] * s
    
    # 2. Agentic Factuality
    statements = decompose_report(report)
    labels = []
    for stmt in statements:
        evidence = retrieve_evidence(stmt, query)
        labels.append(verify_consistency(stmt, evidence)) # RIGHT/WRONG/CONFLICT/UNKNOWN
    factuality_acc = compute_accuracy(labels)
    
    # 3. Process-Centric
    intrinsic = evaluate_process_dimensions(process_log)
    align = compute_alignment(process_log, report)
    s_process = alpha * intrinsic + (1 - alpha) * align
    
    return s_quality, factuality_acc, s_process

Common pitfalls

• Fixed evaluation criteria fail to capture task-specific nuances; the benchmark requires dynamic rubric generation per query.
• Traditional fact-checking assumes a single evidence source, but this benchmark must handle conflicting evidence from both web searches and uploaded attachments.
• Process quality is often ignored in favor of final report quality, but here it is explicitly audited via structural decomposition and alignment checks.

Evidence (verbatim from paper)

The evaluator assesses the report R against each criterion: s_{d,c}=LLM_θ(R, d, c, Q), s_{d,c}∈[0,10], and the final quality score is computed as S_quality=∑{d∈D} W_d ∑{c} w_{d,c} s_{d,c}.

Citation

@misc{ye2026miroeval,
  title={MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome},
  author={Fangda Ye et al. (2026)},
  year={2026},
  note={arXiv:2603.28407}
}

• arXiv: 2603.28407