Detect Goodhart Metric Gaming
When a measure becomes a target it stops being a good measure, because people optimise the shortest path to the number rather than the goal it stood for. Find the shortcut before it finds you.
Procedure
- Write the goal the metric proxies, then list the cheapest ways to move the metric without moving the goal.
- For each shortcut, find the counter-metric that would move the wrong way and add it as a guardrail.
- Inspect the distribution, not the mean — gaming often shows as a new mode, like a spike of rows where
closed_at - opened_at < 60s. - Check the metric definition for a gaming surface: a rate built on a proxy event (page views, replies, sign-ups) invites the cheapest proxy.
- Compare the metric against a slower, harder-to-game outcome (retention, repeat purchase,
reopened_within_7d) and watch for divergence. - Look at the top movers: a few accounts or agents driving the change is a gaming signature, not a system improvement.
- Set the guardrail as a constraint, not a trade-off: the target only counts if the guardrail holds.
- Re-run the analysis one level down (per team, per segment) each quarter; gaming migrates to whatever is now unmeasured.
- Alert on the ratio, not the level — a guardrail degrading while the target improves is the earliest signal.
- When a new shortcut appears, add its counter-metric to the guardrail set rather than arguing about the definition.
Pitfalls
- Per-agent "tickets resolved" incentivises closing early;
reopened_within_7dis the guardrail that catches it. - Lines of code, commits, and story points are all gameable and all get gamed the moment they are reviewed.
- Engagement metrics reward notification spam until uninstalls arrive on a lag.
- One aggregate team target with individual visibility creates the same shortcuts one level down.
- Removing the metric entirely is not the fix; pairing it with a guardrail measures the goal more honestly.
- A target and its guardrail reviewed on different cadences let the guardrail slip for months before anyone notices.
- Averaging over a long window smooths the gaming spike out of view; check weekly, not quarterly.
- A metric that is only ever reviewed, never acted on, still drifts toward the number because people assume it is used.
Verification
psql "$DSN" -c "SELECT agent, count(*) resolved, count(*) FILTER (WHERE reopened_within_7d) reopened FROM tickets GROUP BY 1 ORDER BY 2 DESC LIMIT 10;"
Report: "Resolution time improved 18% but reassignment rose 26% and reopened-within-7d doubled for the top 3 agents — the target is being gamed by premature closure; guardrail added."