# Test The Measuring Stick

> Check whether a score, grade, or ranking actually has teeth before you trust it. Use when relying on any quality number: an AI grading its own or another AI's output, a rubric score on writing, a relevance ranking, a benchmark result. Triggers include "is this quality score real", "does my grader/eval actually work", "can I trust this ranking", "does this benchmark mean anything". Produces a dummy-answer result, a cheap-floor comparison, and a verdict on whether the measure is trustworthy.

- Skill: `ariaxhan/test-the-measuring-stick` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add ariaxhan/test-the-measuring-stick`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ariaxhan/test-the-measuring-stick/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ariaxhan (https://skillmd.com/u/ariaxhan)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ariaxhan/test-the-measuring-stick

---


# Test the Measuring Stick

Before you trust what a score says about the work, test what the score says about garbage. A grader that rates a worthless answer highly is not measuring quality: it is measuring something else, and every "good" score it gave is unverifiable. This finds that out cheaply.

Use it whenever a number is about to drive a decision, especially when an AI assigned the number.

## 1. Build the laziest fake answer

Construct a response that does zero real work but has the right shape:

- Copy the question back as the "answer".
- Return an empty-but-correctly-formatted result.
- Give a single canned, content-free line.

The goal is something that should obviously score near the bottom if the grader works.

## 2. Run the dummy through the exact same scoring

Use the identical grader, rubric, prompt, or ranking, unchanged. No special-casing. Whatever scored the real work scores the dummy.

## 3. Read what the dummy score tells you

- **Dummy scores low** → the stick has teeth. The real scores mean something.
- **Dummy scores high** → the stick is **broken**. A presence/format/length signal is being rewarded instead of quality. The real scores are now unverified, not good. Treat every one of them as suspect.

A format or structure bonus is the usual culprit: it can float an empty answer to the top.

## 4. Check for a saturated ceiling

If almost everything (good, mediocre, dummy) clusters near the top, the test is too easy to separate quality at all. Reaching for a bigger, more expensive grader to push the winners higher is p-hacking, not science. Make the test harder, not the grader stronger.

## 5. Compare against the cheap floor, not zero

"Beats chance" or "beats nothing" is the wrong bar. Name the cheapest thing that already works (the free tool, a one-line baseline, copy-paste, the incumbent) and confirm the expensive option beats **that**. An expensive signal that loses to a free baseline is not an improvement, however high its raw score.

## Output

Return:

- The dummy answer used and the score it received.
- The verdict: stick has teeth / task is trivially easy (saturated ceiling) / stick is broken.
- The cheap-floor comparison: what the cheapest working alternative scores, and whether the expensive option actually beats it.
- If the stick is broken, name the signal it is actually rewarding (presence, format, length) so it can be fixed.