Helpsteer2 Eval

helpsteer2-eval

HelpSteer2: Open-source dataset for training top-performing reward models — Wang et al. (2024) (arXiv:2406.08673, 2024)

What this evaluates

This dataset evaluates language model responses across five key dimensions: helpfulness, correctness, coherence, complexity, and verbosity. It probes an LLM's ability to follow instructions, maintain factual accuracy, produce logically consistent text, and adapt to varying levels of detail and difficulty.

Datasets

• HelpSteer2 — total 21362; splits: train (-1), val (-1)

Metrics

• helpfulness (primary) — range: [0, 4]
  • Annotators rate the response on a Likert-5 scale from 0 to 4 based on how well it addresses the user's prompt.
• correctness — range: [0, 4]
  • Annotators rate factual accuracy and adherence to instructions on a Likert-5 scale from 0 to 4.
• coherence — range: [0, 4]
  • Annotators rate logical consistency and flow on a Likert-5 scale from 0 to 4.
• complexity — range: [0, 4]
  • Annotators rate the difficulty of the prompt/response on a Likert-5 scale from 0 to 4.
• verbosity — range: [0, 4]
  • Annotators rate the length and detail level of the response on a Likert-5 scale from 0 to 4.

Input / output format

Input: User prompt (single-turn or multi-turn conversation history).

Output: Model-generated response text.

Scoring recipe

def aggregate_annotations(annotations):
    # annotations: list of dicts with 'annotator_id' and attribute scores (0-4)
    if len(annotations) < 3:
        return None
    # Retain annotations from the three most agreeing annotators
    agreeing = get_top_3_agreeing(annotations)
    # Filter: keep sample only if max(helpfulness) - min(helpfulness) <= 2
    if max(a['helpfulness'] for a in agreeing) - min(a['helpfulness'] for a in agreeing) > 2:
        return None
    # Final metric value is the mean of the retained scores
    return mean(a['helpfulness'] for a in agreeing)

Common pitfalls

• Initial inter-annotator agreement is low (Cohen's κ ≈ 0.465) and requires iterative guideline clarification and vendor QA to reach acceptable levels (κ ≈ 0.706).
• The 2-point threshold for helpfulness disagreement is used to filter samples, which removes ~10% of data but balances annotation noise vs. inherent subjectivity.
• Multi-turn prompts replace original assistant turns with in-house model generations to avoid restrictive licenses, which may affect response style distribution compared to raw user data.

Evidence (verbatim from paper)

Specifically, for each response, we annotate five attributes (helpfulness, correctness, coherence, complexity, and verbosity) on a Likert-5 scale. As a final step, we retained only responses for which the differences in helpfulness attribute among annotators were 2 points or below on a Likert-5 scale (for both responses to a common prompt), resulting in the removal of about 10% of the samples.

Citation

@misc{wang2024helpsteer2,
  title={HelpSteer2: Open-source dataset for training top-performing reward models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.08673}
}

• arXiv: 2406.08673