Memcollab Eval

memcollab-eval

MemCollab: Cross-Agent Memory Collaboration via Contrastive Trajectory Distillation — Chang et al. (2026) (arXiv:2603.23234, 2026)

What this evaluates

Evaluates LLM agents' ability to solve mathematical reasoning and code generation tasks by leveraging a shared, contrastively distilled memory system. It probes cross-agent knowledge transfer, reasoning invariance extraction, and task-aware memory retrieval efficiency.

Datasets

• MATH500 — total 1500; splits: memory (1000), test (500)
• GSM8K — total 1500; splits: memory (1000), test (500)
• MBPP — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly solved instances out of the total evaluated instances.
• Average Accuracy (%) — range: percent
  • Mean of Accuracy (%) across MATH500, GSM8K, MBPP, and HumanEval.
• Average reasoning turns — range: other
  • Mean number of reasoning turns required to solve an instance.

Input / output format

Input: Task prompts for mathematical reasoning or code generation problems.

Output: Model-generated reasoning trajectories and final answers.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if is_correct(p, g))
    return (correct / len(gold)) * 100

Common pitfalls

• Data leakage: The 1000 instances used to construct the memory system must be strictly disjoint from the 500 instances used for evaluation.
• Naive memory transfer: Directly transferring memory from a single larger model can degrade performance compared to vanilla baselines; contrastive construction is required.
• Retrieval budget: Performance degrades if more than 3 memory entries are retrieved due to noise and attention dispersion.

Evidence (verbatim from paper)

From each dataset, we randomly sample 1000 instances to construct the memory system and evaluate performance on a disjoint set of 500 randomly selected instances, reporting accuracy as the metric*(kang2025distilling). For code generation, we evaluate on MBPP(austin2021program)* and HumanEval*(chen2021evaluating)*.

Citation

@misc{chang2026memcollab,
  title={MemCollab: Cross-Agent Memory Collaboration via Contrastive Trajectory Distillation},
  author={Chang et al. (2026)},
  year={2026},
  note={arXiv:2603.23234}
}

• arXiv: 2603.23234