Es Memeval Eval

es-memeval-eval

ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support — Chen et al. (2026) (arXiv:2602.01885, 2026)

What this evaluates

This benchmark evaluates conversational agents' long-term memory capabilities in personalized emotional support contexts. It probes five core competencies—information extraction, temporal reasoning, conflict detection, abstention, and user modeling—across question answering, summarization, and dialogue generation tasks.

Datasets

• ES-MemEval — total 1368; splits: QA (1209), Summarization (125), Dialogue Generation (34); repo https://github.com/slptongji/ES-MemEval

Metrics

• F1-Score (primary) — range: [0, 1]
  • Token-level F1 overlap between the model's generated answer and the reference answer.
• BERTScore — range: [0, 1]
  • Semantic similarity computed using contextual embeddings between the model response and reference answer.
• LLM-as-Judge (primary) — range: [0, 5]
  • GPT-4o scores semantic consistency and faithfulness on a scale of 0-2 for QA, 0-5 for summarization, and 5-point for dialogue generation based on provided prompts.
• ROUGE-1/2/L — range: [0, 1]
  • Lexical overlap metrics measuring unigram, bigram, and longest common subsequence similarity between generated and reference summaries.
• Event-based F1 — range: [0, 1]
  • F1 score computed on discrete events extracted from both reference and generated summaries to assess factual coverage alignment.
• LLM Rating — range: [0, 5]
  • GPT-4o rates overall dialogue quality on a 5-point scale across long-term memory, personalization, and emotional support dimensions.

Input / output format

Input: Varies by task: (1) QA: question text, reference answer, and supporting evidence passage; (2) Summarization: thematic groups of sessions and event timelines; (3) Dialogue Generation: scenario overview, specific details, user physical/psychological state, and relevant prior sessions.

Output: Task-specific: (1) QA: natural language answer; (2) Summarization: cross-session summary text; (3) Dialogue Generation: model's conversational response turn.

Scoring recipe

def score_qa(pred, ref, question):
    f1 = token_f1(pred, ref)
    bert = bertscore(pred, ref)
    llm_judge = gpt4o_score_semantic(question, ref, pred, scale=0-2)
    return f1, bert, llm_judge

def score_sum(pred, ref):
    rouge = rouge_score(pred, ref, ['ROUGE-1','ROUGE-2','ROUGE-L'])
    llm_judge = gpt4o_score_faithfulness(ref, pred, scale=0-5)
    events_ref = extract_events(ref)
    events_pred = extract_events(pred)
    event_f1 = f1_score(events_ref, events_pred)
    return rouge, llm_judge, event_f1

def score_dialogue(pred_history, scenario):
    llm_rating = gpt4o_rate_quality(pred_history, scale=5)
    obs_recall = observation_recall(pred_history, scenario_annotations)
    return llm_rating, obs_recall

Common pitfalls

• LLM-as-Judge scores are highly sensitive to prompt phrasing and temperature, making cross-run comparisons difficult without strict prompt locking.
• Event-based metrics require consistent event extraction; minor paraphrasing in summaries can artificially lower precision/recall scores.
• Models frequently fail to abstain when information is insufficient, generating hallucinated responses that negatively impact QA and dialogue metrics.

Evidence (verbatim from paper)

Answer quality is assessed using F1-Score (Rajpurkar et al., [2016]), BERTScore (Zhang et al., [2020]), and LLM-as-Judge (Zheng et al., [2023]). F1-Score measures token-level overlap with the reference answer, while BERTScore computes semantic similarity through contextual embeddings. In addition, LLM-as-Judge enables flexible evaluation of model responses: GPT-4o receives the question, reference answer, and model response, and assigns a score of 0, 1, or 2 reflecting semantic consistency.

Citation

@misc{chen2026esmemeval,
  title={ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2602.01885}
}

• arXiv: 2602.01885