# Reflectsumm Eval

> reflectsumm-eval

- Skill: `qhjqhj00/reflectsumm-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/reflectsumm-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/reflectsumm-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/reflectsumm-eval

---


# reflectsumm-eval

> ReflectSumm: A Benchmark for Course Reflection Summarization — Zhong et al. (2024) (arXiv:2403.19012, 2024)

## What this evaluates

Evaluates NLP models on summarizing student course reflections across three tasks: document selection, phrase extraction with support counts, and abstractive summarization. It probes specificity-aware summarization capabilities and model robustness in low-resource educational settings with variable text structure.

## Datasets

- **ReflectSumm** — total 17512; splits: train (-1), val (-1), test (-1); repo https://github.com/EngSalem/ReflectSUMM

## Metrics

- `Unspecified in text` **(primary)** — range: percent
  - Standard summarization metrics (e.g., ROUGE, BERTScore) or human evaluation are implied for benchmarking, but exact formulas and units are not provided in the section.

## Input / output format

**Input**: A collection of student reflection documents from a single lecture, optionally augmented with 4-point specificity scores per reflection.

**Output**: Three task variants: (1) exactly 5 selected reflection documents, (2) exactly 5 phrases each with an accompanying support count, (3) a single abstractive summary of approximately 40 words.

## Scoring recipe

```python
# Extractive: Compute exact match or overlap between predicted 5 docs and gold 5 docs
# Phrase: Compute phrase-level overlap (e.g., ROUGE) between predicted 5 phrases+counts and gold
# Abstractive: Compute sentence/document-level overlap (e.g., ROUGE-L) between predicted ~40-word summary and gold
# Note: Exact metric implementation not specified in text; standard summarization metrics apply.
```

## Common pitfalls

- Models struggle with variable reflection depth and structure, leading to poor content coherence and topic relevance.
- Ignoring specificity scores when prompted, which degrades performance on specificity-aware tasks.
- Failing to respect exact output constraints (e.g., selecting exactly 5 items or generating ~40 words).

## Evidence (verbatim from paper)

> Corresponding to the human extractive summary task, the goal of our models is to pinpoint the five most salient reflections (documents) from a collection of reflections within the same lecture. Human annotators were given the task of summarizing students’ reflections concisely and coherently within ~40 words.

## Citation

```bibtex
@misc{zhong2024reflectsumm,
  title={ReflectSumm: A Benchmark for Course Reflection Summarization},
  author={Zhong et al. (2024)},
  year={2024},
  note={arXiv:2403.19012}
}
```

- arXiv: 2403.19012