Honest Metrics Audit

Honest Metrics Audit

Install

Save this file as ~/.claude/skills/honest-metrics-audit/SKILL.md, or .claude/skills/honest-metrics-audit/SKILL.md to scope it to one repo. Claude Code auto-discovers it. Invoke with /honest-metrics-audit or by asking "do the numbers on this page actually mean what they say?".

Why this exists

Nobody sets out to publish a false number. It happens because a label and a query drift apart.

Someone computes items.filter(i => i.seller.verified).length, which is a count of ITEMS. Someone else renders it under the label "Verified sellers". Both lines are individually reasonable. Together they tell every visitor there are fifteen verified sellers when there is one. No test fails, because no test knows what the label says.

This is worse than an ordinary bug in one specific way: it is indistinguishable from lying. A visitor who discovers it does not conclude you made a mistake. And for anything that looks like traction (users, downloads, revenue, reviews) an overstated number is the single fastest way to lose the trust you were trying to build with it.

Step 1: Inventory every displayed number

Find the components that render figures:

rg -n "toLocaleString|formatCount|formatStat|Intl\.NumberFormat|CountUp|\.length\}" \
  --type-add 'ui:*.{tsx,jsx,vue,svelte}' -t ui

Also find the labels themselves, which is often faster:

rg -n "(users|customers|sellers|buyers|members|downloads|installs|reviews|ratings|sales|revenue|active|verified|trusted)" \
  -i --type-add 'ui:*.{tsx,jsx,vue,svelte}' -t ui

Build a table. One row per number a real visitor can see:

Do not skip numbers that seem obviously fine. The mislabeled ones always seem obviously fine.

Step 2: For each row, name the unit

This is the whole audit. For every number, answer in one sentence: what is one of the things being counted?

Then compare that answer to the noun in the label. Mismatches to look for:

• Wrong entity. Counting items but labelling people. Counting rows but labelling distinct users. Counting events but labelling sessions.
• Missing distinct. orders.length labelled "customers" when one customer can order twice.
• Wrong population. A count that includes test accounts, internal staff, soft-deleted rows, or your own first-party listings, under a label implying third parties.
• Wrong window. "Active users" computed with no time bound. "This month" computed from a rolling 30 days. "Today" computed in the wrong timezone.
• Wrong denominator. A percentage whose base excludes the cases most likely to fail, which flatters the rate.

Write the corrected label OR the corrected expression for every mismatch. Both are valid fixes; choose whichever makes the number more useful.

Step 3: Hunt fabricated and placeholder figures

rg -n "(stars|downloads|users|reviews|rating|count)\s*[:=]\s*[0-9]{2,}" -t code
rg -n "\b(1,?234|4\.[89]|99\.9|10,?000\+|trusted by|join [0-9])" -i -t code

Flag any number that is a literal rather than a query result, anywhere it can reach a user. Then trace whether it can:

• Is it in a seed or fixture file that a production command could run?
• Is it a fallback value used when the real query fails?
• Is it a default prop that renders when data is missing?

A hardcoded 4.9 rating with no reviews behind it is the most damaging thing in this category, and it usually arrives as design placeholder text that nobody removed.

Step 4: Check the zero and failure paths

Two questions per number:

What renders when the true value is zero? A brand-new product showing "0 downloads" is honest but reads as dead. Showing nothing is also honest. What is not acceptable is a fallback that substitutes a non-zero placeholder. Find the zero handling and say which of the three it does.

What renders when the query fails? Trace the error path. If a failed lookup falls back to a cached, default, or hardcoded figure, the UI will assert a number that is not merely stale but unsupported. A stat that fails closed (hidden) is correct. A stat that fails to a plausible-looking default is a defect.

rg -n "\?\?\s*[0-9]+|\|\|\s*[0-9]+|catch.*return\s*[0-9]+" -t code

Step 5: Check aggregate claims in prose

Numbers hide in sentences, not just in stat blocks:

rg -n "(over|more than|nearly|join|trusted by|used by|thousands|millions)" \
  -i --type-add 'ui:*.{tsx,jsx,vue,svelte,md,mdx}' -t ui

Marketing copy is rarely wired to a query at all. For each claim, find the evidence or mark it unsupported. "Trusted by thousands of developers" in a hardcoded string is a claim your code cannot back.

Step 6: Report

[SEVERITY] <label as shown to the user>
  Shows:    <the expression>
  Counts:   <what one unit actually is>
  Claims:   <what the label says it is>
  Visible:  <the route a real visitor sees it on>
  Fix:      <corrected label, or corrected expression>

Severity:

• CRITICAL the number overstates reality to a visitor right now, or is fabricated. Overstatement is worse than understatement; rank it that way.
• HIGH correct today but will overstate under a reachable condition (a second seller, a failed query, a zero case).
• MEDIUM ambiguous or unlabelled units, or a claim in prose with no source.

End with a count: how many displayed numbers you traced, and how many matched their labels. If everything is honest, that pair of numbers is the evidence.

Rules

• Trace every number to the expression that produces it. A label that "looks right" is not verified.
• Never fix an overstatement by making the label vaguer. "Community" instead of "sellers" hides the problem rather than correcting it.
• Never propose inventing or padding a number to make a screen look healthier. If a real number is unflattering, the honest options are to show it, to show nothing, or to show a different real number.
• Understating is a bug too, just a cheaper one. Report it, rank it lower.

From Toolbay. Free to use, modify, and share. Keep this line and others can find it too.