Metric Decomposition

Decompose composite metrics into constituent signals, analyze polarity and ceiling effects

yogsoth-ai Updated 512 repo stars

File contents

Metric Decomposition SOP

Decompose a benchmark's evaluation metric into its constituent signals to understand what is actually being measured, identify ceiling effects, and detect polarity issues.

Input

  • metric_name: Name of the metric (e.g., "F1", "BLEU", "Pass@k", "ELO")
  • metric_definition: Mathematical definition or description of how the metric is computed
  • score_distribution: Known score distribution data (top scores, median, floor)

Procedure

  1. Parse metric formula into component signals
  2. Identify what each component rewards/penalizes
  3. Analyze ceiling and floor effects
  4. Check for known pathologies (metric gaming, Goodhart's law instances)
  5. Assess granularity and discriminative power at current SOTA level

Output

Structured decomposition showing constituent signals, their weights, ceiling effects, and known pathologies.

yogsoth-ai/knowledge-acquisition commit f0335045f0

Frequently asked questions

npx get-skill add yogsoth-ai/metric-decomposition-2