Seer Eval

seer-eval

SEER: The Span-based Emotion Evidence Retrieval Benchmark — Sampath et al. (2025) (arXiv:2510.03490, 2025)

What this evaluates

Evaluates LLMs' ability to identify precise textual spans expressing emotion within single-sentence and multi-sentence contexts, distinguishing emotion evidence from other linguistic elements.

Datasets

• SEER — total 1200; splits: test (-1); repo https://github.com/chailab-umich/SEER

Metrics

• F1 (primary) — range: [0, 1]
  • Span-level F1 score computed over retrieved or highlighted emotion evidence spans. Exact match is required for highlighting; hallucinations score 0.
• Cosine Similarity (Sim) — range: [0, 1]
  • Cosine similarity between the embedding of the predicted span and the gold span.

Input / output format

Input: Single-sentence or multi-sentence text passages containing potential emotion evidence.

Output: Retrieved span text or highlighted text with verbatim markup indicating the emotion evidence.

Scoring recipe

def evaluate(predictions, golds):
    f1s, sims = [], []
    for pred, gold in zip(predictions, golds):
        f1s.append(compute_span_f1(pred, gold))
        sims.append(cosine_similarity(embed(pred), embed(gold)))
    return mean(f1s), mean(sims)

Common pitfalls

• CoT prompting degrades performance on single-sentence tasks but improves it on multi-sentence tasks.
• Highlight prompt is significantly harder than Retrieve due to strict verbatim requirements; any hallucination yields a score of 0.
• Model size does not directly predict performance; smaller models can outperform much larger ones.

Evidence (verbatim from paper)

Table 3: F1 and cosine similarity (Sim) scores for Task 1 (Retrieve and Highlight). Each entry is averaged over five runs with standard deviations.

Citation

@misc{sampath2025seer,
  title={SEER: The Span-based Emotion Evidence Retrieval Benchmark},
  author={Sampath et al. (2025)},
  year={2025},
  note={arXiv:2510.03490}
}

• arXiv: 2510.03490