yogsoth-ai Contamination Audit SOP
Detect evidence of train-test data leakage, benchmark contamination, and memorization artifacts that inflate reported scores beyond genuine capability.
Input
- benchmark_name: Name of the benchmark being audited
- training_data_sources: Known or suspected training data sources for models evaluated on this benchmark
- test_set_metadata: Information about the test set (size, creation date, source, public availability)
Procedure
- Assess temporal contamination risk (test data predates training cutoff?)
- Check for known contamination disclosures in model papers
- Search for contamination studies specific to this benchmark
- Analyze performance patterns indicative of memorization
- Assess mitigation measures (canary strings, held-out splits, version rotation)
Output
Contamination risk assessment with evidence and confidence levels.
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use |
|---|---|
| spawn-agent | Spawn a customized CC subagent with full MCP tool access. Used by SOPs that declare execution: subagent. |