analyze-leaderboard-dynamics
Purpose
Analyze leaderboard score distributions, compression or saturation, anomalies, selective reporting, and regime shifts.
Input contract
required: [leaderboard_records, metric_schema, time_or_version_field]
optional: [evaluation_protocols, source_quality, method_metadata]
constraints: [scores require task, metric, condition, source, and observation provenance]
Procedure
- Normalize scores and evaluation conditions into comparable groups.
- Inspect distributions, rank movement, compression, and missingness.
- Identify anomalies, selective reporting, and regime changes.
- State whether additional evidence is continuing, near-saturated, or saturated.
Output contract
produces: [leaderboard_distribution, regime_segments, anomaly_report, saturation_assessment]
delta_fields: [findings, evidence_updates, uncertainties, decisions]
Quality gates
- Every reported regime or anomaly must cite the records and comparison basis.
- Do not compare scores across incompatible protocols without an explicit limitation.
Failure and counterexamples
Do not treat rank order as progress when metric definitions or evaluation conditions changed; preserve ties and missing observations.
Provenance map
resolved: leaderboard-dynamics-analysis
1---2name: analyze-leaderboard-dynamics3description: "Analyze leaderboard score distributions, compression/saturation, anomalies, selective reporting, and regime shifts."4metadata:5 source: https://skillmd.com/skills/yogsoth-ai/analyze-leaderboard-dynamics6---78# analyze-leaderboard-dynamics910## Purpose1112Analyze leaderboard score distributions, compression or saturation, anomalies, selective reporting, and regime shifts.1314## Input contract1516```yaml17required: [leaderboard_records, metric_schema, time_or_version_field]18optional: [evaluation_protocols, source_quality, method_metadata]19constraints: [scores require task, metric, condition, source, and observation provenance]20```2122## Procedure23241. Normalize scores and evaluation conditions into comparable groups.252. Inspect distributions, rank movement, compression, and missingness.263. Identify anomalies, selective reporting, and regime changes.274. State whether additional evidence is continuing, near-saturated, or saturated.2829## Output contract3031```yaml32produces: [leaderboard_distribution, regime_segments, anomaly_report, saturation_assessment]33delta_fields: [findings, evidence_updates, uncertainties, decisions]34```3536## Quality gates3738- Every reported regime or anomaly must cite the records and comparison basis.39- Do not compare scores across incompatible protocols without an explicit limitation.4041## Failure and counterexamples4243Do not treat rank order as progress when metric definitions or evaluation conditions changed; preserve ties and missing observations.4445## Provenance map4647- `resolved: leaderboard-dynamics-analysis`