# Castle Eval

> This benchmark evaluates large language models' ability to detect and mitigate educational safety risks in a student-tailored manner. It probes how well models adapt their responses to individual student profiles across 15 risk domains and 14 psychological and educational attributes. Use when the user wants to benchmark on CASTLE, or asks about evaluating this task. Reports safety score.

- Skill: `qhjqhj00/castle-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/castle-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/castle-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/castle-eval

---


# castle-eval

> CASTLE: A Comprehensive Benchmark for Evaluating Student-Tailored Personalized Safety in Large Language Models — Jia et al. (2026) (arXiv:2602.05633, 2026)

## What this evaluates

This benchmark evaluates large language models' ability to detect and mitigate educational safety risks in a student-tailored manner. It probes how well models adapt their responses to individual student profiles across 15 risk domains and 14 psychological and educational attributes.

## Datasets

- **CASTLE** — total 92908; splits: test (-1)

## Metrics

- `safety score` **(primary)** — range: [0, 5]
  - A 5-point scale rating the safety and appropriateness of the model's response to a student query, considering the 15 educational risk domains and student profile context. Higher scores indicate better risk detection and mitigation.

## Input / output format

**Input**: Student query in Chinese or English, optionally accompanied by a student profile (non-personalized, implicit hints, or explicit attributes).

**Output**: Natural language response addressing the student query.

## Scoring recipe

```python
def compute_safety_score(predictions, queries, profiles):
    scores = []
    for pred, q, p in zip(predictions, queries, profiles):
        # Judge evaluates response safety on 1-5 scale
        # across 15 educational risk domains given profile p
        score = evaluate_safety(pred, q, p)
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Evaluating models in non-personalized settings yields consistently low scores (1.7–2.2), which may underestimate their true safety capabilities when context is provided.
- Chinese queries consistently score higher than English ones due to higher information density per token, potentially skewing cross-lingual safety comparisons.
- Assuming that adding student profiles will fully eliminate safety risks; the benchmark shows scores improve but never reach perfection across all dimensions.

## Evidence (verbatim from paper)

> Table 2 reports the safety score (5-point scale) for each model under non-personalized settings, from which we derive the following key observations. ... Specifically, scores increase from an initial range of 1.8-2.5 to above 3.0 in multiple domains, with the largest improvements observed in psychological well-being and academic integrity.

## Citation

```bibtex
@misc{jia2026castle,
  title={CASTLE: A Comprehensive Benchmark for Evaluating Student-Tailored Personalized Safety in Large Language Models},
  author={Jia et al. (2026)},
  year={2026},
  note={arXiv:2602.05633}
}
```

- arXiv: 2602.05633