yogsoth-ai Construct Validity Assessment SOP
Evaluate whether a benchmark actually measures the capability it claims to measure, using psychometric validity frameworks adapted for AI evaluation.
Input
- benchmark_name: Name of the benchmark
- claimed_capability: What the benchmark authors claim it measures
- task_examples: Representative examples from the benchmark
Procedure
- Define the construct (claimed capability) precisely
- Analyze task requirements — what skills are actually needed to solve examples?
- Assess content validity — do items representatively sample the construct?
- Check convergent validity — correlation with other measures of same construct
- Check discriminant validity — independence from unrelated constructs
- Identify construct-irrelevant variance (confounds)
Output
Validity verdict with evidence for each validity dimension.
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use |
|---|---|
| spawn-agent | Spawn a customized CC subagent with full MCP tool access. Used by SOPs that declare execution: subagent. |