Egomem Eval

egomem-eval

EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models — Yao et al. (2025) (arXiv:2509.11914, 2025)

What this evaluates

Evaluates a lifelong memory agent's ability to perform real-time audiovisual user retrieval, detect dialog session boundaries in continuous streams, and generate personalized, fact-consistent responses in full-duplex omnimodal interactions.

Datasets

• LFW — total ?; splits: test (-1); repo https://github.com/serengil/deepface/tree/master/benchmarks
• VoxCeleb — total ?; splits: test (-1)
• EgoMem Custom Text Retrieval — total 200; splits: test (200)
• EgoMem Episodic Trigger — total 1000; splits: test (1000)

Metrics

• Accuracy — range: [0, 1]
  • Fraction of correctly verified face pairs out of total test pairs.
• pass@1 — range: [0, 1]
  • Binary indicator: 1 if the top-1 retrieved speaker matches the query speaker, else 0.
• EER — range: [0, 1]
  • Equal Error Rate: the decision threshold where false acceptance rate equals false rejection rate.
• pass@5 (primary) — range: [0, 1]
  • Fraction of queries where all relevant facts are returned within the top-5 retrieved entries.
• Jaccard score — range: [0, 1]
  • Intersection over union of predicted and ground-truth dialog session spans.
• span_match@N — range: [0, 1]
  • Precision, recall, and F1 for detected boundaries allowing ±N step tolerance from ground truth.
• Fact Score (primary) — range: [0, 1]
  • Binary 0/1 score from an LLM judge indicating if the response is personalized, consistent with the user profile, and factually correct.
• Answer Quality — range: [0, 10]
  • LLM-assigned score from 0 to 10 measuring general helpfulness and quality independent of personalization.

Input / output format

Input: Retrieval: query (face image, speech utterance, or text keyword) vs candidate pool. Episodic Trigger: continuous token stream segments. Personalized Dialog: user instruction, ground-truth response, MemChunks contents, and generated response (for evaluator).

Output: Retrieval: ranked candidate list or binary match decision. Episodic Trigger: detected dialog boundary timestamps. Personalized Dialog: generated textual/audio response.

Scoring recipe

def compute_pass_at5(retrieved_top5, gold_facts):
    return 1.0 if all(f in retrieved_top5 for f in gold_facts) else 0.0

def compute_span_match_f1(predicted_spans, gold_spans, tolerance=5):
    tp = sum(1 for p in predicted_spans if any(abs(p - g) <= tolerance for g in gold_spans))
    precision = tp / len(predicted_spans) if predicted_spans else 0
    recall = tp / len(gold_spans) if gold_spans else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0

def compute_fact_score(prediction, user_profile, evaluator_model):
    prompt = f"Check if '{prediction}' matches profile: {user_profile}. Return 1 if consistent, 0 otherwise."
    return 1 if evaluator_model(prompt) == "1" else 0

Common pitfalls

• Speaker verification EER threshold is manually tuned to 6 for deployment based on human case studies rather than using a fixed benchmark threshold.
• Episodic trigger span_match@0 degrades significantly in noisy environments due to inherent VAD latency, which is expected but often misinterpreted as a model failure.
• Fact Score relies on an external LLM judge (DeepSeek-V3), introducing non-determinism and prompt sensitivity that can affect reproducibility.

Evidence (verbatim from paper)

For each dialog turn, we provide an evaluator model with the following inputs: the user instruction (textual transcript), the ground-truth textual response, the contents of the MemChunks, and the textual monologue response generated by RoboEgo. The evaluator is implemented with the DeepSeek-V3 API, prompted to return two scores: – Fact Score: A binary 0/1 metric for each turn indicating whether the model’s response is personalized to the user and consistent with the user profile, without factual errors. – Answer Quality: A score from 0 to 10 for each turn measuring the general helpfulness and quality of the response with respect to the user instruction, independent of personalization.

Citation

@misc{yao2025egomem,
  title={EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models},
  author={Yao et al. (2025)},
  year={2025},
  note={arXiv:2509.11914}
}

• arXiv: 2509.11914