yogsoth-ai Metric Decomposition SOP
Decompose a benchmark's evaluation metric into its constituent signals to understand what is actually being measured, identify ceiling effects, and detect polarity issues.
Input
- metric_name: Name of the metric (e.g., "F1", "BLEU", "Pass@k", "ELO")
- metric_definition: Mathematical definition or description of how the metric is computed
- score_distribution: Known score distribution data (top scores, median, floor)
Procedure
- Parse metric formula into component signals
- Identify what each component rewards/penalizes
- Analyze ceiling and floor effects
- Check for known pathologies (metric gaming, Goodhart's law instances)
- Assess granularity and discriminative power at current SOTA level
Output
Structured decomposition showing constituent signals, their weights, ceiling effects, and known pathologies.
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use |
|---|---|
| spawn-agent | Spawn a customized CC subagent with full MCP tool access. Used by SOPs that declare execution: subagent. |