# Helpsteer2 Eval

> helpsteer2-eval

- Skill: `qhjqhj00/helpsteer2-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/helpsteer2-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/helpsteer2-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/helpsteer2-eval

---


# helpsteer2-eval

> HelpSteer2: Open-source dataset for training top-performing reward models — Wang et al. (2024) (arXiv:2406.08673, 2024)

## What this evaluates

This dataset evaluates language model responses across five key dimensions: helpfulness, correctness, coherence, complexity, and verbosity. It probes an LLM's ability to follow instructions, maintain factual accuracy, produce logically consistent text, and adapt to varying levels of detail and difficulty.

## Datasets

- **HelpSteer2** — total 21362; splits: train (-1), val (-1)

## Metrics

- `helpfulness` **(primary)** — range: [0, 4]
  - Annotators rate the response on a Likert-5 scale from 0 to 4 based on how well it addresses the user's prompt.
- `correctness` — range: [0, 4]
  - Annotators rate factual accuracy and adherence to instructions on a Likert-5 scale from 0 to 4.
- `coherence` — range: [0, 4]
  - Annotators rate logical consistency and flow on a Likert-5 scale from 0 to 4.
- `complexity` — range: [0, 4]
  - Annotators rate the difficulty of the prompt/response on a Likert-5 scale from 0 to 4.
- `verbosity` — range: [0, 4]
  - Annotators rate the length and detail level of the response on a Likert-5 scale from 0 to 4.

## Input / output format

**Input**: User prompt (single-turn or multi-turn conversation history).

**Output**: Model-generated response text.

## Scoring recipe

```python
def aggregate_annotations(annotations):
    # annotations: list of dicts with 'annotator_id' and attribute scores (0-4)
    if len(annotations) < 3:
        return None
    # Retain annotations from the three most agreeing annotators
    agreeing = get_top_3_agreeing(annotations)
    # Filter: keep sample only if max(helpfulness) - min(helpfulness) <= 2
    if max(a['helpfulness'] for a in agreeing) - min(a['helpfulness'] for a in agreeing) > 2:
        return None
    # Final metric value is the mean of the retained scores
    return mean(a['helpfulness'] for a in agreeing)
```

## Common pitfalls

- Initial inter-annotator agreement is low (Cohen's κ ≈ 0.465) and requires iterative guideline clarification and vendor QA to reach acceptable levels (κ ≈ 0.706).
- The 2-point threshold for helpfulness disagreement is used to filter samples, which removes ~10% of data but balances annotation noise vs. inherent subjectivity.
- Multi-turn prompts replace original assistant turns with in-house model generations to avoid restrictive licenses, which may affect response style distribution compared to raw user data.

## Evidence (verbatim from paper)

> Specifically, for each response, we annotate five attributes (helpfulness, correctness, coherence, complexity, and verbosity) on a Likert-5 scale. As a final step, we retained only responses for which the differences in helpfulness attribute among annotators were 2 points or below on a Likert-5 scale (for both responses to a common prompt), resulting in the removal of about 10% of the samples.

## Citation

```bibtex
@misc{wang2024helpsteer2,
  title={HelpSteer2: Open-source dataset for training top-performing reward models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.08673}
}
```

- arXiv: 2406.08673