Locomo10 Eval

locomo10-eval

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents — Cao et al. (2026) (arXiv:2604.18349, 2026)

What this evaluates

This benchmark evaluates long-term conversational memory systems by testing their ability to retrieve relevant dialogue turns and answer questions over extended, multi-session histories. It probes semantic reasoning, temporal tracking, and adversarial robustness across five distinct question categories.

Datasets

• LoCoMo10 — total 10; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Standard question-answering F1 score, computed as the harmonic mean of token-level precision and recall between the predicted answer and the gold answer.
• Precision@K — range: [0, 1]
  • The fraction of retrieved turns that match gold evidence turns. Computed at the question level and macro-averaged over all questions.
• Recall@K — range: [0, 1]
  • The fraction of gold evidence turns recovered by the retrieved set. Computed at the question level and macro-averaged over all questions.
• Avg K — range: other
  • The average number of turns in the final evidence set provided to the answer-generation LLM per question.

Input / output format

Input: A long multi-session dialogue history (average 587 turns) and a natural language question requiring evidence from the conversation.

Output: A generated answer string, plus the set of retrieved dialogue turns used as evidence for retrieval metric computation.

Scoring recipe

def score_instance(retrieved_turns, gold_turns, pred_ans, gold_ans):
    matches = len(set(retrieved_turns) & set(gold_turns))
    prec_k = matches / max(len(retrieved_turns), 1)
    rec_k = matches / max(len(gold_turns), 1)
    f1 = qa_f1_score(pred_ans, gold_ans)
    return {'F1': f1, 'Precision@K': prec_k, 'Recall@K': rec_k}
# Aggregate: macro-average all per-instance scores across the dataset

Common pitfalls

• Pooling retrieved and gold evidence turns across all questions before computing Precision@K/Recall@K, which violates the explicit instruction to compute them at the question level and macro-average.
• Assuming a higher Avg K (context size) automatically improves answer quality, whereas the benchmark emphasizes retrieving a compact, precise evidence set to reduce context cost.
• Comparing hierarchical retrieval methods against flat baselines without accounting for the different retrieval budgets or LLM reasoning steps required.

Evidence (verbatim from paper)

Precision@K is the fraction of retrieved turns that match gold evidence turns, while Recall@K is the fraction of gold evidence turns recovered by the final evidence set. Since a question may require one or more gold evidence turns, we compute Precision@K and Recall@K at the question level and report their macro-average over questions, rather than pooling evidence turns across all questions.

Citation

@misc{cao2026higemem,
  title={HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents},
  author={Cao et al. (2026)},
  year={2026},
  note={arXiv:2604.18349}
}

• arXiv: 2604.18349