# Vietmed Sum Eval

> Evaluates abstractive summarization capabilities on real-world and simulated medical conversations in Vietnamese, testing both human-transcribed and ASR-generated noisy transcripts. Use when the user wants to benchmark on VietMed-Sum, or asks about evaluating this task. Reports ROUGE.

- Skill: `qhjqhj00/vietmed-sum-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/vietmed-sum-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/vietmed-sum-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/vietmed-sum-eval

---


# vietmed-sum-eval

> Real-time Speech Summarization for Medical Conversations — Le-Duc et al. (2024) (arXiv:2406.15888, 2024)

## What this evaluates

Evaluates abstractive summarization capabilities on real-world and simulated medical conversations in Vietnamese, testing both human-transcribed and ASR-generated noisy transcripts.

## Datasets

- **VietMed-Sum** — total ?; splits: train (-1), test (-1); repo https://github.com/leduckhai/MultiMed

## Metrics

- `ROUGE` **(primary)** — range: [0, 1]
  - Measures n-gram overlap between generated summaries and reference summaries. The paper evaluates both local and global summaries using standard ROUGE variants.

## Input / output format

**Input**: Transcribed speech text (either human-transcribed or ASR-generated) representing medical conversation utterances.

**Output**: Abstractive summary text (generated locally after N utterances and globally at conversation end).

## Scoring recipe

```python
def compute_rouge(predictions, references):
    scores = {}
    for metric in ['ROUGE-1', 'ROUGE-2', 'ROUGE-L']:
        metric_scores = [rouge_score(ref, pred, use_stemmer=True) for ref, pred in zip(references, predictions)]
        scores[metric] = sum(metric_scores) / len(metric_scores)
    return scores
```

## Common pitfalls

- The paper evaluates on both human transcripts and ASR transcripts, but does not specify if ROUGE scores are reported separately for each or averaged.
- ROUGE variants (ROUGE-1, ROUGE-2, ROUGE-L) are not explicitly detailed in the main text; readers must check the Appendix for exact configuration.
- Local vs. global summary evaluation is mentioned, but the exact split or aggregation method for these two types of summaries is not fully specified in the provided section.

## Evidence (verbatim from paper)

> We use ROUGE *[[14]]*, a metric commonly used for summarization, to evaluate our models. To evaluate their performance, we calculate their ROUGE scores on the local and global summaries in the test set.

## Citation

```bibtex
@misc{leduc2024realtimespeechsummarization,
  title={Real-time Speech Summarization for Medical Conversations},
  author={Le-Duc et al. (2024)},
  year={2024},
  note={arXiv:2406.15888}
}
```

- arXiv: 2406.15888