How do I install the Test The Measuring Stick skill?

Run npx skillmds@latest add ariaxhan/test-the-measuring-stick in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Test The Measuring Stick safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Test The Measuring Stick?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Test The Measuring Stick free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Test The Measuring Stick?

ariaxhan (@ariaxhan) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Test The Measuring Stick by ariaxhan

Name: Test The Measuring Stick
Author: ariaxhan

Test The Measuring Stick

Check whether a score, grade, or ranking actually has teeth before you trust it. Use when relying on any quality number: an AI grading its own or another AI's output, a rubric score on writing, a relevance ranking, a benchmark result. Triggers include "is this quality score real", "does my grader/eval actually work", "can I trust this ranking", "does this benchmark mean anything". Produces a dummy-answer result, a cheap-floor comparison, and a verdict on whether the measure is trustworthy.

ariaxhan Updated Sep 17, 2026

File contents

Test the Measuring Stick

Before you trust what a score says about the work, test what the score says about garbage. A grader that rates a worthless answer highly is not measuring quality: it is measuring something else, and every "good" score it gave is unverifiable. This finds that out cheaply.

Use it whenever a number is about to drive a decision, especially when an AI assigned the number.

1. Build the laziest fake answer

Construct a response that does zero real work but has the right shape:

Copy the question back as the "answer".
Return an empty-but-correctly-formatted result.
Give a single canned, content-free line.

The goal is something that should obviously score near the bottom if the grader works.

2. Run the dummy through the exact same scoring

Use the identical grader, rubric, prompt, or ranking, unchanged. No special-casing. Whatever scored the real work scores the dummy.

3. Read what the dummy score tells you

Dummy scores low → the stick has teeth. The real scores mean something.
Dummy scores high → the stick is broken. A presence/format/length signal is being rewarded instead of quality. The real scores are now unverified, not good. Treat every one of them as suspect.

A format or structure bonus is the usual culprit: it can float an empty answer to the top.

4. Check for a saturated ceiling

If almost everything (good, mediocre, dummy) clusters near the top, the test is too easy to separate quality at all. Reaching for a bigger, more expensive grader to push the winners higher is p-hacking, not science. Make the test harder, not the grader stronger.

5. Compare against the cheap floor, not zero

"Beats chance" or "beats nothing" is the wrong bar. Name the cheapest thing that already works (the free tool, a one-line baseline, copy-paste, the incumbent) and confirm the expensive option beats that. An expensive signal that loses to a free baseline is not an improvement, however high its raw score.

Output

Return:

The dummy answer used and the score it received.
The verdict: stick has teeth / task is trivially easy (saturated ceiling) / stick is broken.
The cheap-floor comparison: what the cheapest working alternative scores, and whether the expensive option actually beats it.
If the stick is broken, name the signal it is actually rewarding (presence, format, length) so it can be fixed.

---name: test-the-measuring-stickdescription: Check whether a score, grade, or ranking actually has teeth before you trust it. Use when relying on any quality number: an AI grading its own or another AI's output, a rubric score on writing, a relevance ranking, a benchmark result. Triggers include "is this quality score real", "does my grader/eval actually work", "can I trust this ranking", "does this benchmark mean anything". Produces a dummy-answer result, a cheap-floor comparison, and a verdict on whether the measure is trustworthy.---# Test the Measuring StickBefore you trust what a score says about the work, test what the score says about garbage. A grader that rates a worthless answer highly is not measuring quality: it is measuring something else, and every "good" score it gave is unverifiable. This finds that out cheaply.Use it whenever a number is about to drive a decision, especially when an AI assigned the number.## 1. Build the laziest fake answerConstruct a response that does zero real work but has the right shape:- Copy the question back as the "answer".- Return an empty-but-correctly-formatted result.- Give a single canned, content-free line.The goal is something that should obviously score near the bottom if the grader works.## 2. Run the dummy through the exact same scoringUse the identical grader, rubric, prompt, or ranking, unchanged. No special-casing. Whatever scored the real work scores the dummy.## 3. Read what the dummy score tells you- **Dummy scores low** → the stick has teeth. The real scores mean something.- **Dummy scores high** → the stick is **broken**. A presence/format/length signal is being rewarded instead of quality. The real scores are now unverified, not good. Treat every one of them as suspect.A format or structure bonus is the usual culprit: it can float an empty answer to the top.## 4. Check for a saturated ceilingIf almost everything (good, mediocre, dummy) clusters near the top, the test is too easy to separate quality at all. Reaching for a bigger, more expensive grader to push the winners higher is p-hacking, not science. Make the test harder, not the grader stronger.## 5. Compare against the cheap floor, not zero"Beats chance" or "beats nothing" is the wrong bar. Name the cheapest thing that already works (the free tool, a one-line baseline, copy-paste, the incumbent) and confirm the expensive option beats **that**. An expensive signal that loses to a free baseline is not an improvement, however high its raw score.## OutputReturn:- The dummy answer used and the score it received.- The verdict: stick has teeth / task is trivially easy (saturated ceiling) / stick is broken.- The cheap-floor comparison: what the cheapest working alternative scores, and whether the expensive option actually beats it.- If the stick is broken, name the signal it is actually rewarding (presence, format, length) so it can be fixed.

Frequently asked questions