qhjqhj00 childsafe-safety-eval
Evaluating LLM Safety Across Child Development Stages: A Simulated Agent Approach — Murali et al. (2025) (arXiv:2510.05484, 2025)
What this evaluates
Evaluates LLM safety alignment across four child developmental stages (ages 6–17) using simulated agents grounded in developmental psychology. It probes how models handle sensitive contexts, boundary-testing, and age-specific cognitive limitations in multi-turn interactions.
Datasets
- ChildSafe Dataset — total 1200; splits: test (1200)
Metrics
semantic_safety_score(primary) — range: [0, 1]- LLM-as-Judge scoring based on predefined rubrics across nine safety dimensions (e.g., privacy, misinformation, emotional support). Scores are averaged across dimensions and conversation traces.
Input / output format
Input: Multi-turn conversation prompts initialized with age-specific system prompts simulating child cognitive profiles (ages 6–8, 9–11, 12–14, 15–17) across sensitive and neutral contexts.
Output: Model-generated text responses per turn in the multi-turn dialogue.
Scoring recipe
scores = []
for trace in dataset:
responses = model.generate(trace.prompts, T=0.7, top_p=0.9, max_tokens=1024)
judge_score = judge.evaluate(responses, rubrics) # GPT-4o, T=0.0
scores.append(judge_score)
return mean(scores)
Common pitfalls
- Using adult-centric benchmarks fails to capture developmental vulnerabilities like over-reliance on AI or misunderstanding figurative language.
- Inference temperature significantly impacts safety behavior; non-zero T (0.7) may increase refusal rates or unsafe outputs compared to T=0.
- LLM-as-Judge determinism requires strict temperature control (T=0.0) to avoid scoring variance across runs.
Evidence (verbatim from paper)
The LLM-as-Judge ($\mathcal{J}$) utilized for the Semantic Scoring (Section 3.3) was instantiated using GPT-4o, configured with temperature $T=0.0$ to maximize determinism in scoring. ... The complete dataset of 1,200 annotated conversation traces.
Citation
@misc{murali2025childsafe,
title={Evaluating LLM Safety Across Child Development Stages: A Simulated Agent Approach},
author={Murali et al. (2025)},
year={2025},
note={arXiv:2510.05484}
}
- arXiv: 2510.05484