Zenbrain Memory Eval

zenbrain-memory-eval

ZenBrain: A Neuroscience-Inspired 7-Layer Memory Architecture for Autonomous AI Systems — Bering (2026) (arXiv:2604.23878, 2026)

What this evaluates

This evaluation protocol assesses the long-term memory and retrieval capabilities of autonomous AI systems. It measures how well models retain, route, and retrieve information across multiple sessions and varying context lengths, while also evaluating the quality of generated answers using LLM-as-a-judge scoring.

Datasets

• LoCoMo (Real-LoCoMo pool) — total 1986; splits: test (1986)
• LongMemEval-S — total 500; splits: test (500)
• MemoryAgentBench — total ?; splits: (unstated)
• MemoryArena — total ?; splits: (unstated)

Metrics

• NDCG@5 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5, measuring the quality of the top-5 retrieved documents relative to an ideal ranking. Calculated as DCG@5 / IDCG@5, where DCG discounts relevance by log2(rank+2).
• P@5 — range: [0, 1]
  • Precision at rank 5, measuring the proportion of relevant documents in the top-5 retrieved results.
• R@5 — range: [0, 1]
  • Recall at rank 5, measuring the fraction of all relevant documents found in the top-5 results.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank, the inverse of the rank of the first relevant document in the retrieved list.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall at rank 5.
• LLM-as-Judge — range: [0, 1]
  • Normalized mean score (0–5 rubric) assigned by three independent LLM judges to the generated answer, rescaled to [0, 1]. Binary-thresholded at ≥3 for significance testing.

Input / output format

Input: Query text paired with a context window containing stored facts/haystacks (shared across queries for LoCoMo, per-question isolated for LongMemEval-S).

Output: Top-5 retrieved documents and a generated answer to the query.

Scoring recipe

def compute_ndcg_at_k(retrieved_docs, relevant_docs, k=5):
    dcg = sum(1 / math.log2(i + 2) for i, doc in enumerate(retrieved_docs[:k]) if doc in relevant_docs)
    idcg = sum(1 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
    return dcg / idcg if idcg > 0 else 0.0

def compute_llm_judge_score(generated_answer, query, rubric="0-5"):
    score = llm_judge.evaluate(query, generated_answer, rubric)
    return score / 5.0  # Rescale to [0, 1]

# Aggregate across dataset
ndcg_mean = np.mean([compute_ndcg_at_k(preds, gold) for preds, gold in zip(predictions, golds)])
judge_mean = np.mean([compute_llm_judge_score(ans, q) for ans, q in zip(answers, queries)])

Common pitfalls

• Embedder quality differences can confound ranking metrics; the protocol mandates sharing a single embedding backbone (nomic-embed-text) for competitive runs.
• LLM judge scores may exhibit provider-specific bias; the protocol requires cross-provider agreement checks and seed-averaging to mitigate this.
• System failures (e.g., HTTP 500 errors) can create selection bias if only successful queries are averaged; the protocol scores all queries, treating failures as zero-retrieval cases.

Evidence (verbatim from paper)

On raw retrieval, mem0 tops P@5/R@5/F1 via a permissive recall budget while letta wins MRR/NDCG@5.

Citation

@misc{berin2026zenbrain,
  title={ZenBrain: A Neuroscience-Inspired 7-Layer Memory Architecture for Autonomous AI Systems},
  author={Bering (2026)},
  year={2026},
  note={arXiv:2604.23878}
}

• arXiv: 2604.23878