Locomo Eval

locomo-eval

Evaluating Very Long-Term Conversational Memory of LLM Agents — Maharana et al. (2024) (arXiv:2402.17753, 2024)

What this evaluates

This benchmark evaluates the long-term conversational memory of LLM agents by testing their ability to answer questions, summarize events, and generate multi-modal dialogues over very long, multi-session conversations. It probes how well models retain and reason over temporal and causal information across hundreds of turns and thousands of tokens.

Datasets

• LoCoMo — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Standard F1 score computed between predicted answer text and ground truth answer text, calculated as the harmonic mean of precision and recall.
• Recall Accuracy (R@$k$) — range: [0, 1]
  • Measures whether the correct retrieval unit (dialog, observation, or summary) appears in the top-k retrieved items.

Input / output format

Input: Dialogue history (text-only with image captions for QA/summarization, or raw images for multi-modal generation) interleaved with a question or prompt.

Output: Predicted answer text (for QA) or generated dialogue turn (for multi-modal generation).

Scoring recipe

def compute_f1(pred, gold):
    pred_tokens = set(pred.lower().split())
    gold_tokens = set(gold.lower().split())
    if not pred_tokens or not gold_tokens:
        return 0.0
    precision = len(pred_tokens & gold_tokens) / len(pred_tokens)
    recall = len(pred_tokens & gold_tokens) / len(gold_tokens)
    if precision + recall == 0:
        return 0.0
    return 2 * precision * recall / (precision + recall)

Common pitfalls

• Models are evaluated under different context length constraints (e.g., 4K, 8K, 16K), so results are not directly comparable without accounting for the truncated history.
• RAG experiments use different retrieval units (dialog, observation, summary) and top-k values, making it difficult to isolate the effect of the retriever versus the reader model.

Evidence (verbatim from paper)

Results are based on F1-score for answer prediction; higher is better.

Citation

@misc{maharana2024locomo,
  title={Evaluating Very Long-Term Conversational Memory of LLM Agents},
  author={Maharana et al. (2024)},
  year={2024},
  note={arXiv:2402.17753}
}

• arXiv: 2402.17753