# Locomo10 Eval

> This benchmark evaluates long-term conversational memory systems by testing their ability to retrieve relevant dialogue turns and answer questions over extended, multi-session histories. It probes semantic reasoning, temporal tracking, and adversarial robustness across five distinct question categories. Use when the user wants to benchmark on LoCoMo10, or asks about evaluating this task. Reports F1 score.

- Skill: `qhjqhj00/locomo10-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/locomo10-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/locomo10-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/locomo10-eval

---


# locomo10-eval

> HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents — Cao et al. (2026) (arXiv:2604.18349, 2026)

## What this evaluates

This benchmark evaluates long-term conversational memory systems by testing their ability to retrieve relevant dialogue turns and answer questions over extended, multi-session histories. It probes semantic reasoning, temporal tracking, and adversarial robustness across five distinct question categories.

## Datasets

- **LoCoMo10** — total 10; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Standard question-answering F1 score, computed as the harmonic mean of token-level precision and recall between the predicted answer and the gold answer.
- `Precision@K` — range: [0, 1]
  - The fraction of retrieved turns that match gold evidence turns. Computed at the question level and macro-averaged over all questions.
- `Recall@K` — range: [0, 1]
  - The fraction of gold evidence turns recovered by the retrieved set. Computed at the question level and macro-averaged over all questions.
- `Avg K` — range: other
  - The average number of turns in the final evidence set provided to the answer-generation LLM per question.

## Input / output format

**Input**: A long multi-session dialogue history (average 587 turns) and a natural language question requiring evidence from the conversation.

**Output**: A generated answer string, plus the set of retrieved dialogue turns used as evidence for retrieval metric computation.

## Scoring recipe

```python
def score_instance(retrieved_turns, gold_turns, pred_ans, gold_ans):
    matches = len(set(retrieved_turns) & set(gold_turns))
    prec_k = matches / max(len(retrieved_turns), 1)
    rec_k = matches / max(len(gold_turns), 1)
    f1 = qa_f1_score(pred_ans, gold_ans)
    return {'F1': f1, 'Precision@K': prec_k, 'Recall@K': rec_k}
# Aggregate: macro-average all per-instance scores across the dataset
```

## Common pitfalls

- Pooling retrieved and gold evidence turns across all questions before computing Precision@K/Recall@K, which violates the explicit instruction to compute them at the question level and macro-average.
- Assuming a higher Avg K (context size) automatically improves answer quality, whereas the benchmark emphasizes retrieving a compact, precise evidence set to reduce context cost.
- Comparing hierarchical retrieval methods against flat baselines without accounting for the different retrieval budgets or LLM reasoning steps required.

## Evidence (verbatim from paper)

> Precision@K is the fraction of retrieved turns that match gold evidence turns, while Recall@K is the fraction of gold evidence turns recovered by the final evidence set. Since a question may require one or more gold evidence turns, we compute Precision@K and Recall@K at the question level and report their macro-average over questions, rather than pooling evidence turns across all questions.

## Citation

```bibtex
@misc{cao2026higemem,
  title={HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents},
  author={Cao et al. (2026)},
  year={2026},
  note={arXiv:2604.18349}
}
```

- arXiv: 2604.18349