Ms Tod Eval

ms-tod-eval

MemGuide: Intent-Driven Memory Selection for Goal-Oriented Multi-Session LLM Agents — Du et al. (2025) (arXiv:2505.20231, 2025)

What this evaluates

Evaluates an LLM agent's ability to retrieve and utilize long-term memory across multiple dialogue sessions to complete goal-oriented tasks. It probes intent-aligned memory selection, slot-level tracking, and dialogue efficiency in maintaining task continuity over extended interactions.

Datasets

• MS-TOD — total ?; splits: test (-1)
• SGD — total ?; splits: test (-1)
• MultiWOZ 2.2 — total ?; splits: test (-1)

Metrics

• Success Rate (S.R.) (primary) — range: [0, 1]
  • Binary indicator: 1 if the user's final goal is explicitly confirmed as achieved in the dialogue, 0 otherwise.
• Joint Goal Accuracy (JGA) — range: [0, 1]
  • Exact-match accuracy of predicted slot values against gold-standard slot-value sets across all dialogue turns.
• Dialogue Turn Efficiency (DTE) — range: turns
  • Average number of dialogue turns required to complete a task.
• GPT-4 score — range: [1, 10]
  • 1–10 rating assigned by GPT-4-as-a-judge evaluating response fluency, coherence, and informativeness.

Input / output format

Input: Dialogue history, current user utterance, and retrieved memory units (intent-aligned QA pairs or raw history).

Output: Model-generated dialogue response (text).

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions: list of model responses per turn
    # gold: dict with 'goal', 'slots', 'confirmed'
    sr = 1.0 if gold['confirmed'] else 0.0
    jga = 0.0
    for turn in predictions:
        predicted_slots = extract_slots(turn)
        if predicted_slots == gold['slots']:
            jga += 1.0
    jga /= len(predictions)
    dte = len(predictions)
    return {'S.R.': sr, 'JGA': jga, 'DTE': dte}

Common pitfalls

• Evaluating on non-confirmation-type sessions inflates dialogue length and deflates success rate; only sessions with explicit goal confirmation should be included.
• JGA requires exact slot-value matching; partial matches or synonym variations are typically counted as incorrect unless explicitly normalized.
• GPT-4 scoring is sensitive to prompt formatting and temperature; results may vary if the judge prompt differs from the appendix version.

Evidence (verbatim from paper)

We use four core automatic metrics and human evaluation to evaluate response performance: 1) GPT-4 score, (1–10) 333GPT-4-as-the-judge prompts can be found in Appendix evaluates response quality in terms of fluency, coherence, and informativeness; 2) Joint Goal Accuracy (JGA) measures slot prediction accuracy; 3) Dialogue Turn Efficiency (DTE) captures the number of turns required to complete a task, and 4) Success Rate (S.R.) indicates whether the user goal is achieved.

Citation

@misc{du2025memguide,
  title={MemGuide: Intent-Driven Memory Selection for Goal-Oriented Multi-Session LLM Agents},
  author={Du et al. (2025)},
  year={2025},
  note={arXiv:2505.20231}
}

• arXiv: 2505.20231