# Harness Metrics Skill

> Procedure for deterministic metric collection, paired degradation tests, safety gates, fertility, and Pareto analysis.

- Skill: `kevin-liu-01/harness-metrics-skill` (Agent Skill, multi-file: 7 files)
- Install (CLI): `npx skillmds@latest add kevin-liu-01/harness-metrics-skill`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kevin-liu-01/harness-metrics-skill/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: kevin-liu-01 (https://skillmd.com/u/kevin-liu-01)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/kevin-liu-01/harness-metrics-skill

---


# cotcodec / harness/metrics

## Purpose
<!-- agent-docs:fill:purpose -->

Metrics turn raw trace events into auditable per-cell and cross-condition
measurements for success, cost, latency, safety, fidelity, and degradation.

## Mental model & key files
<!-- agent-docs:fill:model -->

- `collector.py` owns per-step observations.
- `analyzer.py` aggregates conditions and Pareto frontiers.
- `degradation.py` owns exact-key paired comparisons and McNemar analysis.
- `fertility.py` measures tokenizer fragmentation; `safety.py` owns red-line gates.

## Patterns to follow / invariants
<!-- agent-docs:fill:patterns -->

- Collect all inexpensive raw metrics before aggregation.
- Pair on immutable task/seed/model keys; reject duplicates and missing mates.
- Report denominators, confidence/uncertainty, and incomplete cells explicitly.
- Do not impute failed or missing runs as successes.
- Safety failures remain first-class outcomes and can kill an otherwise efficient arm.

## Common tasks → first action
<!-- agent-docs:fill:tasks -->

- Metric change: update collector schema, analyzer, fixtures, and backward-compat tests.
- Degradation audit: verify exact pairing and discordant-cell counts before p-values.
- New safety rule: preregister threshold and baseline comparison before execution.

## Gotchas
<!-- agent-docs:fill:gotchas -->

- Statistical significance cannot rescue protocol-invalid or incomplete evidence.
- Tokenizer fertility, billed tokens, and retained context bytes are different measures.