Reasoning Collab Memory Eval

reasoning-collab-memory-eval

Enhancing Reasoning with Collaboration and Memory — Michelman et al. (2025) (arXiv:2503.05944, 2025)

What this evaluates

Evaluates LLM logical and spatial reasoning capabilities under multi-agent collaboration and memory-augmented prompting. It probes how different reasoning styles, exemplar retrieval methods, and answer aggregation strategies impact accuracy on formal logic and object-tracking tasks.

Datasets

• FOLIO — total ?; splits: test (-1)
• RACO — total ?; splits: test (-1)
• TSO — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy computed after converting model outputs to a canonical form. For multi-agent setups, the final answer is determined by majority voting or a summarizer agent, then compared to the ground truth label.

Input / output format

Input: Natural language or formal logical prompts describing scenes, object states, or logical premises, optionally augmented with few-shot exemplars retrieved from a memory bank.

Output: Canonicalized answer string (e.g., 'True', 'False', 'Unknown', or specific object/color/position/count). Multi-agent pipelines aggregate outputs via voting or a summarizer LLM call.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    canonical_preds = [convert_to_canonical(p) for p in predictions]
    correct = sum(1 for p, g in zip(canonical_preds, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Outputs must be strictly converted to a canonical form before comparison; minor formatting differences cause false negatives.
• Multi-agent voting requires consistent answer formatting across all agents to avoid split votes.
• Memory retrieval method (similarity vs. random) significantly impacts performance, contrary to typical assumptions about semantic search.

Evidence (verbatim from paper)

LLM answers are converted to a canonical form before comparing to the label or counting votes for multi-agent collaboration. FOLIO (Han et al., [2022]) is a first-order logic task. It presents a series of formal logical statements then asks whether the conclusion is True, False, or Unknown. Reasoning About Colored Objects (RACO) from BIG-bench (Srivastava et al., [2022]) describes a scene involving several objects of various colors, optionally makes a modification, then asks about colors, positions, or counts.

Citation

@misc{michelman2025enhancing,
  title={Enhancing Reasoning with Collaboration and Memory},
  author={Michelman et al. (2025)},
  year={2025},
  note={arXiv:2503.05944}
}

• arXiv: 2503.05944