Marca Eval

marca-eval

MARCA: A Checklist-Based Benchmark for Multilingual Web Search — Almeida et al. (2026) (arXiv:2604.14448, 2026)

What this evaluates

Evaluates LLMs' ability to perform multilingual web search and extract multiple entities from search results. It probes task decomposition, cross-lingual retrieval, and evidence aggregation under different agentic interaction frameworks.

Datasets

• MARCA — total 52; splits: test (52); repo https://github.com/maritaca-ai/MARCA

Metrics

• Checklist Accuracy (primary) — range: [0, 1]
  • The fraction of ground-truth checklist items (entities/facts) correctly identified and attributed in the model's final answer relative to the total number of items in the checklist. Computed per question and averaged across the dataset.

Input / output format

Input: A multi-entity information-seeking question in English or Portuguese. The model interacts with a web search engine either directly (Basic framework) or via an orchestrator delegating to subagents (Orchestrator framework).

Output: A final natural language answer containing a list of entities/facts that should match the predefined checklist for the question.

Scoring recipe

def compute_checklist_accuracy(predictions, gold_checklists):
    correct = 0
    total = 0
    for pred, checklist in zip(predictions, gold_checklists):
        pred_entities = extract_entities_from_answer(pred)
        matched = len(set(pred_entities) & set(checklist))
        correct += matched
        total += len(checklist)
    return correct / total

Common pitfalls

• Models may implicitly translate Portuguese queries to English, missing region-specific content that only exists in Portuguese.
• Orchestration overhead can hurt performance for strong models that already handle complex queries in a single context.
• Checklist accuracy measures coverage/completeness, not necessarily factual correctness of individual retrieved snippets.

Evidence (verbatim from paper)

Table 1 reports Checklist Accuracy (mean ± run-level standard deviation) across languages and inference frameworks for 14 models.

Citation

@misc{almeida2026marca,
  title={MARCA: A Checklist-Based Benchmark for Multilingual Web Search},
  author={Almeida et al. (2026)},
  year={2026},
  note={arXiv:2604.14448}
}

• arXiv: 2604.14448