Reflectsumm Eval

reflectsumm-eval

ReflectSumm: A Benchmark for Course Reflection Summarization — Zhong et al. (2024) (arXiv:2403.19012, 2024)

What this evaluates

Evaluates NLP models on summarizing student course reflections across three tasks: document selection, phrase extraction with support counts, and abstractive summarization. It probes specificity-aware summarization capabilities and model robustness in low-resource educational settings with variable text structure.

Datasets

• ReflectSumm — total 17512; splits: train (-1), val (-1), test (-1); repo https://github.com/EngSalem/ReflectSUMM

Metrics

• Unspecified in text (primary) — range: percent
  • Standard summarization metrics (e.g., ROUGE, BERTScore) or human evaluation are implied for benchmarking, but exact formulas and units are not provided in the section.

Input / output format

Input: A collection of student reflection documents from a single lecture, optionally augmented with 4-point specificity scores per reflection.

Output: Three task variants: (1) exactly 5 selected reflection documents, (2) exactly 5 phrases each with an accompanying support count, (3) a single abstractive summary of approximately 40 words.

Scoring recipe

# Extractive: Compute exact match or overlap between predicted 5 docs and gold 5 docs
# Phrase: Compute phrase-level overlap (e.g., ROUGE) between predicted 5 phrases+counts and gold
# Abstractive: Compute sentence/document-level overlap (e.g., ROUGE-L) between predicted ~40-word summary and gold
# Note: Exact metric implementation not specified in text; standard summarization metrics apply.

Common pitfalls

• Models struggle with variable reflection depth and structure, leading to poor content coherence and topic relevance.
• Ignoring specificity scores when prompted, which degrades performance on specificity-aware tasks.
• Failing to respect exact output constraints (e.g., selecting exactly 5 items or generating ~40 words).

Evidence (verbatim from paper)

Corresponding to the human extractive summary task, the goal of our models is to pinpoint the five most salient reflections (documents) from a collection of reflections within the same lecture. Human annotators were given the task of summarizing students’ reflections concisely and coherently within ~40 words.

Citation

@misc{zhong2024reflectsumm,
  title={ReflectSumm: A Benchmark for Course Reflection Summarization},
  author={Zhong et al. (2024)},
  year={2024},
  note={arXiv:2403.19012}
}

• arXiv: 2403.19012