vamseeachanta Magnitude Checks (+2)
Magnitude Checks
For any key number in your analysis, verify it passes the "smell test":
| Metric Type | Sanity Check |
|---|---|
| User counts | Does this match known MAU/DAU figures? |
| Revenue | Is this in the right order of magnitude vs. known ARR? |
| Conversion rates | Is this between 0% and 100%? Does it match dashboard figures? |
| Growth rates | Is 50%+ MoM growth realistic, or is there a data issue? |
| Averages | Is the average reasonable given what you know about the distribution? |
| Percentages | Do segment percentages sum to ~100%? |
Cross-Validation Techniques
- Calculate the same metric two different ways and verify they match
- Spot-check individual records -- pick a few specific entities and trace their data manually
- Compare to known benchmarks -- match against published dashboards, finance reports, or prior analyses
- Reverse engineer -- if total revenue is X, does per-user revenue times user count approximately equal X?
- Boundary checks -- what happens when you filter to a single day, a single user, or a single category? Are those micro-results sensible?
Red Flags That Warrant Investigation
- Any metric that changed by more than 50% period-over-period without an obvious cause
- Counts or sums that are exact round numbers (suggests a filter or default value issue)
- Rates exactly at 0% or 100% (may indicate incomplete data)
- Results that perfectly confirm the hypothesis (reality is usually messier)
- Identical values across time periods or segments (suggests the query is ignoring a dimension)