pproenca Metric Validation Harness
Point this harness at a candidate metric and a corpus, and it runs experiments that try to falsify each property a trustworthy, optimizable metric must have. It is the empirical companion to deterministic-metric-design: that skill tells you to prove monotonicity, invariance, determinism, and construct validity; this skill runs the experiment and reports PASS/FAIL, each result mapped to the design-skill category it checks.
Read-only. It computes and reports; it never modifies your metric, the corpus, or any external state. Safe to run unsupervised.
When to Apply
- Someone proposes, reviews, tunes, or ships a metric / score / index and you need evidence it is sound
- A score "feels off" — you suspect it tracks LOC, jumps between runs, or saturates
- You are about to let an agent optimize a metric and need to know it can't be gamed by cosmetic edits
- You built a candidate per
deterministic-metric-designand want to empirically confirm the properties you argued for - You are choosing between two metrics and need to know which actually predicts the outcome (and beats a trivial baseline)
Workflow Overview
config.json / env → resolve metric_cmd, corpus, thresholds (env > config > bundled default)
│
▼
verify.sh ──► determinism ─ invariance ─ monotonicity ─ robustness ─ tractability ─ validity
│ (each property check maps to a deterministic-metric-design category)
▼
PASS / FAIL per property → exit 0 (all pass) or 1 (any group failed)
The Adapter Contract
Your metric is any command that takes a path as its last argument and prints exactly one number to stdout:
$ python3 mymetric.py path/to/file.py
42
Language-agnostic — Python, a shell one-liner, a compiled binary, anything. Diagnostics go to stderr; stdout is the number only. A bundled example metric (scripts/examples/metric_ast_nodes.py, AST-node count) ships so the harness runs out of the box.
How to Run
# 1. Validate the bundled example metric (works with zero setup):
bash scripts/verify.sh
# 2. Validate YOUR metric — set metric_cmd in config.json, or override per-run:
METRIC_CMD="python3 /abs/path/mymetric.py" bash scripts/verify.sh
# 3. Prove the harness itself works (positive + negative cases):
bash scripts/selftest.sh
# 4. Sanity-check your adapter prints one number:
bash scripts/run-metric.sh path/to/file.py
verify.sh runs every check and prints a final PASS/FAIL. Each check is also runnable on its own (e.g. bash scripts/check-determinism.sh).
What It Checks
| Check | Maps to (design skill) | What it does | PASS condition |
|---|---|---|---|
check-determinism.sh |
det- |
Runs the metric twice + under PYTHONHASHSEED 0/1 |
identical number every time |
check-invariance.sh |
prop- / game- |
Adds comments/blank lines/whitespace (cosmetic) | score unchanged (else it's gameable) |
check-monotonicity.sh |
prop- |
Appends a code block (construct-increasing) + checks spread | score non-decreasing; not saturated |
check-robustness.sh |
prop- |
Empty + single-statement edge inputs | finite, in declared range, no crash |
check-tractability.py |
comp- |
Times the metric on growing inputs | within budget, sub-quadratic growth |
check-validity.py |
valid- |
Spearman vs accepted, vs LOC; AUC vs outcome | convergent high, discriminant not ~LOC, predictive beats baseline |
Statistics (Spearman, AUC/Mann–Whitney) are pure Python stdlib — no numpy/scipy.
Setup & Configuration
The harness runs with zero config against the bundled example. To validate your own metric, set fields in config.json (or override any of them with the matching UPPER_CASE environment variable per run):
| config.json | Env override | Meaning |
|---|---|---|
metric_cmd |
METRIC_CMD |
your metric command (path-printing → number) |
baseline_cmd |
BASELINE_CMD |
trivial baseline (default: bundled LOC) |
corpus_dir |
CORPUS_DIR |
artifacts the property checks iterate over |
labels_csv |
LABELS_CSV |
path[,outcome][,accepted] for validity |
declared_min / declared_max |
DECLARED_MIN / DECLARED_MAX |
range the robustness check enforces |
Validity thresholds are env-tunable: CONVERGENT_MIN, DISCRIMINANT_MAX, PREDICTIVE_MIN (defaults are lenient — tighten for a real run; see gotchas.md).
Empty config fields fall back to the bundled demo, so the skill never crashes on missing setup — it runs the example instead.
Tool Requirements
python3(3.8+) — runs the metric, the transforms, and the statsbashandawk— the orchestrator and numeric comparisons (scripts are macOS bash 3.2-safe)
No network, no external packages.
Interpreting Results
A FAIL names the property and the design-skill rule to consult. Examples:
- cosmetic noise moved the score → the metric reads surface text; see
prop-prove-invariance-under-irrelevant-transformsandgame-make-cheapest-improvement-the-right-one. - score DROPPED after adding code → non-monotonic; optimizing it can reward worse code (
prop-prove-monotonicity). - |Spearman(metric, LOC)| too high → it's LOC relabeled (
valid-discriminant-not-just-loc).
Related Skills
deterministic-metric-design— the design half. Use it to construct the metric (define the construct, choose a computable proxy, pick the scale, argue the properties); use this harness to empirically verify what you argued.same-results-less-code,complexity-optimizer,knip-deadcode— prescriptive code-reduction skills; validate any reduction metric you build to drive them with this harness before letting an agent optimize against it.
See references/workflow.md for per-check details, how to wire up your own metric and corpus, and troubleshooting.