Carmem Eval

carmem-eval

CarMem: Enhancing Long-Term Memory in LLM Voice Assistants through Category-Bounding — Kirmayr et al. (2025) (arXiv:2501.09645, 2025)

What this evaluates

Evaluates an LLM's ability to extract, maintain, and retrieve long-term user preferences in an in-car voice assistant context using a predefined category-bound schema. It probes structured information extraction, state maintenance via function calling, and semantic retrieval accuracy under privacy-preserving constraints.

Datasets

• CarMem — total 1000; splits: val (500), test (500); repo https://github.com/johanneskirmayr/CarMem

Metrics

• F1-score (primary) — range: [0, 1]
  • Micro-averaged harmonic mean of precision and recall across main, sub, and detail category levels. An extraction is correct only if the predicted main, sub, and detail categories exactly match the ground-truth categories.
• Function-call accuracy — range: [0, 1]
  • Percentage of maintenance utterances where the model correctly calls the ground-truth function (pass, update, or append) based on the utterance type and existing preference state.
• Top-k retrieval accuracy — range: [0, 1]
  • Proportion of retrieval utterances where the ground-truth preference appears in the top-k retrieved results. The parameter k is dynamically set to the number of stored preferences for the specific user and subcategory.

Input / output format

Input: Multi-turn conversational transcripts (Extraction Conversations, Maintenance Utterances, Retrieval Utterances) paired with a predefined category-bound schema (main, sub, detail categories) and existing user preferences.

Output: Structured JSON matching the extraction schema; function calls (pass, update, append) for maintenance; ranked list of retrieved preferences for retrieval.

Scoring recipe

def score_extraction(pred, gold):
    return int(pred.main == gold.main and pred.sub == gold.sub and pred.detail == gold.detail)

def score_maintenance(pred_func, gold_func):
    return int(pred_func == gold_func)

def score_retrieval(pred_list, gold_pref, k):
    return int(gold_pref in pred_list[:k])

# Aggregate metrics
extract_f1 = micro_f1([score_extraction(p, g) for p, g in extraction_data])
maint_acc = mean([score_maintenance(p, g) for p, g in maint_data])
retrieval_acc = mean([score_retrieval(p, g, k) for p, g, k in retriev_data])

Common pitfalls

• Over-extraction vs. non-extraction trade-off: The paper notes a 6% non-extraction rate but 12-25% over-extraction, highlighting that strict schema adherence can cause false positives in semantically similar categories.
• Dynamic k in retrieval: The top-k parameter is not fixed but adapts to the number of stored preferences per user/subcategory, which differs from standard fixed-k benchmarks and requires careful implementation.
• Maintenance evaluation subset: Maintenance is only evaluated on entries with perfect extraction accuracy, which may overestimate real-world pipeline performance where extraction errors propagate.

Evidence (verbatim from paper)

An extraction is considered correct if the main-, sub-, and detail categories match those of the ground-truth preference. A data point is considered correct if the ground truth maintenance function is called. retrieval is considered optimal if the ground-truth preference is among the top-n_{i,j} retrieved preferences, where n_{i,j} represents the number of preferences stored for user i within subcategory j. Table 6: Top-k accuracy for retrieving the ground-truth preference based on the Retrieval Utterance.

Citation

@misc{kirmayr2025carmem,
  title={CarMem: Enhancing Long-Term Memory in LLM Voice Assistants through Category-Bounding},
  author={Kirmayr et al. (2025)},
  year={2025},
  note={arXiv:2501.09645}
}

• arXiv: 2501.09645