Vietmed Sum Eval

vietmed-sum-eval

Real-time Speech Summarization for Medical Conversations — Le-Duc et al. (2024) (arXiv:2406.15888, 2024)

What this evaluates

Evaluates abstractive summarization capabilities on real-world and simulated medical conversations in Vietnamese, testing both human-transcribed and ASR-generated noisy transcripts.

Datasets

• VietMed-Sum — total ?; splits: train (-1), test (-1); repo https://github.com/leduckhai/MultiMed

Metrics

• ROUGE (primary) — range: [0, 1]
  • Measures n-gram overlap between generated summaries and reference summaries. The paper evaluates both local and global summaries using standard ROUGE variants.

Input / output format

Input: Transcribed speech text (either human-transcribed or ASR-generated) representing medical conversation utterances.

Output: Abstractive summary text (generated locally after N utterances and globally at conversation end).

Scoring recipe

def compute_rouge(predictions, references):
    scores = {}
    for metric in ['ROUGE-1', 'ROUGE-2', 'ROUGE-L']:
        metric_scores = [rouge_score(ref, pred, use_stemmer=True) for ref, pred in zip(references, predictions)]
        scores[metric] = sum(metric_scores) / len(metric_scores)
    return scores

Common pitfalls

• The paper evaluates on both human transcripts and ASR transcripts, but does not specify if ROUGE scores are reported separately for each or averaged.
• ROUGE variants (ROUGE-1, ROUGE-2, ROUGE-L) are not explicitly detailed in the main text; readers must check the Appendix for exact configuration.
• Local vs. global summary evaluation is mentioned, but the exact split or aggregation method for these two types of summaries is not fully specified in the provided section.

Evidence (verbatim from paper)

We use ROUGE [[14]], a metric commonly used for summarization, to evaluate our models. To evaluate their performance, we calculate their ROUGE scores on the local and global summaries in the test set.

Citation

@misc{leduc2024realtimespeechsummarization,
  title={Real-time Speech Summarization for Medical Conversations},
  author={Le-Duc et al. (2024)},
  year={2024},
  note={arXiv:2406.15888}
}

• arXiv: 2406.15888