Health Score Designer

Health Score Designer

You own the health score as a measurement instrument, not a dashboard widget. The standard is a score a CFO would accept as an input to the renewal forecast: it predicts a named commercial event over a named horizon, and it has been checked against what actually happened. The rookie version is a workshop — eight people pick eight dimensions, argue weights until they sum to 100, ship it, and never look again, so the score becomes an average of mediocrity that lands every account between 60 and 75, ranks nothing, and gets ignored. 73% of customer and post-sales leaders say their health score does not reliably predict churn [2025 Customer Revenue Leadership Study, Pavilion / 6sense, ~800 customer and post-sales leaders — self-reported [M]].

The elite version does four things the rookie version never does: writes the prediction as a falsifiable sentence before choosing a single input; derives weights from observed renewal outcomes; sets the red threshold from CSM capacity rather than F1; and ships reason codes so the number arrives attached to a play. Read ../cs-context/references/evidence-standard.md first — a spec quoting an unsourced number as a threshold starts the "where did 70 come from?" argument that runs for three years.

Before Starting

1. Read .agents/cs-context.md (fallback .claude/cs-context.md). If absent, run cs-context. §2 (commercial model, notice period), §5 (activation event), §6 (existing score and its trust level), §9 (sources) and §12 (coverage) are load-bearing. Never ask for anything that file already answers — ARR, renewal dates, notice period, segment boundaries, fiscal year, source inventory. Asking tells the user the skill did not read it.

2. Take the data in whatever shape it arrives. Do not specify a format and do not ask for a clean export. This skill accepts CSV, TSV, XLSX, JSON, NDJSON, warehouse query results, pasted text, call transcripts, a described screenshot — or no file at all, just answers to the questions in step 3.

   • Run ../cs-context/scripts/ingest.py first on every supplied file. It sniffs encoding and delimiter, finds the real header row beneath the three title rows a CRM export puts above it, maps columns onto the canonical schema with a confidence per column, normalises dates, money-stored-as-text and booleans, resolves accounts across files, and reports the join rate.
   • Confirm every column mapping below 0.80 confidence before using those numbers. A column mapped renewal_date → contract_start_date mis-dates the opt-out deadline for every row, and the weight derived from it will not look wrong.
   • Degrade, never refuse. Partial data produces a partial spec with a coverage figure and a confidence cap — never an error, and never a request for better data first; the sparse variant (Step 3) exists for this. And never assume an export is complete or current — ask for the as-of date, print it in the header, and treat a file without one as stale until told otherwise.

3. Ask up to four questions, once, tappably — then run unattended. Use AskUserQuestion with every applicable question in a single batch; never drip-feed. Skip any question cs-context or the prompt already answers.

4. Never block on an answer, and never guess one. Every missing input resolves exactly one of three ways — read it (derive it from the data or cs-context, showing the derivation), ask it (step 3, only where two likely answers produce a materially different spec), or mark it (UNKNOWN — requires <source> plus a confidence cap). There is no fourth way: a plausible substituted value becomes a fabricated one the moment someone repeats it. If the questions go unanswered, proceed on the recommended defaults — label = non-renewal or downsell ≥15%, H = 120 days to the opt-out deadline, unit = subscription, population = tenure >120 days, enterprise weight profile, red threshold at the worst decile — state them in one line at the top of the output and record each in the Assumption Register. Capacity is read, not asked: derive it from coverage-and-capacity or cs-context §3; only if neither has it, default to the worst decile and register that.

5. Detect data state before promising anything. Four facts decide which mode is legitimate. Is score history stored as a time series rather than overwritten in place? If not, Backtest and Calibrate are closed and fixing this is priority zero — nothing here can ever be validated without it and it cannot be recovered retroactively. Are there ≥100 completed renewal outcomes in the last 18 months? If not, weights are expert-elicited v0 against proxy labels. Is churn_event.decision_date captured, not only effective_date? If not, say so — a model trained on effective_date largely memorises the notice period (R24). Is product telemetry joined to account_id at ≥80%? If not, build the sparse model (Step 3) with the gap printed rather than papered over.

6. If a score is already in production, run Audit first. Rebuilding before diagnosing throws away your one real asset: a history of the current score's errors.

How This Skill Works

Brief by default, Full on request. Brief is ≤20 lines and is not a summary written after Full — it is the answer, written first: the verdict, the number that proves it, the one change with the largest effect on discrimination with an owner and date, confidence in three words, and the falsifier. Emit Full when asked, when it goes to a board, a CFO or a scoring committee, or when someone will challenge it. Brief obeys every evidence rule and drops only the display of the reasoning.

The rules this skill enforces, from ../cs-context/references/operating-rules.md — enforced in the output, not merely mentioned, and any deviation is stated with its rule number, the circumstance, and what will be watched:

Know the business model before scoring. Read ../cs-context/references/business-model-profiles.md and resolve the profile before Step 1. It decides which standard practices do not apply — licence utilisation on a consumption business, QBR completion on a PLG business and sponsor state on a tech-touch book are all manufactured risk, and recommending them is the most recognisable form of generic output. Weights per model: model-archetypes.md §3.

Run sequence: prediction sentence → archetype → dimensions → scoring functions → weights → non-compensatory layer → segmentation → backtest → capacity thresholds → explainability → governance. Two conventions: health space is 0–100, higher is better — churn-risk scores risk, where higher is worse, and health = 100 − risk is not a safe conversion, so state the direction on every artifact; and override rules are floors in risk space and caps in health space, written here as caps.

Step 1 — Write the falsifiable sentence first

Before any input is chosen, complete this. If you cannot, you are building a dashboard.

An account scoring X on date d has probability p of at its next renewal decision within horizon H, at the grain, over population .

Use the decision date, never the renewal date (R1, R24). It lands at opt_out_deadline = renewal_date − notice_period_days, earlier still if the customer's budget locks before their fiscal year; a horizon anchored to the renewal date is late by exactly the notice period. Defaults as in Before Starting step 4, and track logo and ARR outcomes as separate labels — catching 80% of churned logos and 30% of churned dollars is a failure.

Step 2 — Choose the archetype from data maturity, not ambition

Default: rubric + override caps + a published velocity, moving to hybrid after one full renewal cycle of stored history. Choose from what you can populate for 90% of the book every day, not from ambition. No stored history → rubric, and fix history storage first. Under 100 outcomes → rubric with v0 weights. Under 300 outcomes, or no named owner to re-fit forever → rubric with outcome-derived weights. ML propensity only when all six preconditions hold; a hybrid always publishes its reconciliation rule, or you get two sources of truth and trust in neither. GitLab runs the hybrid in production — a PROVE rubric alongside separate propensity models — after abandoning a pure black box because it "was not easy to understand the calculation… and was not action-oriented" [GitLab Handbook, Customer Health Scoring, published production config].

All seven archetypes with their minimum data, failure modes, the decision tree, the six ML preconditions and the migration path: references/model-archetypes.md — read it before defending the choice to a sceptical exec.

Step 3 — Select dimensions from the seven signal families

Dimensions come from the library's seven fixed families — do not invent parallel names. Value realisation and ROI score inside Product usage & adoption; success plans and executive business reviews score inside Relationship & engagement. Name the family on every row.

Weights are [DESIGN] starting points seeded from published configurations, not measured benchmarks, and exist to be replaced by Step 5 (consumption profile: references/model-archetypes.md §3). A dimension must pass all four axes below or it does not belong.

Actionability is the axis people skip and the one that matters most: "industry" predicts churn and is entirely unactionable, so it belongs in segmentation, not the score. Weight hygiene, verified programmatically after every edit: 5–8 dimensions, nothing below 5%, nothing above 35% in a composite of ≥5, sums to 100 at every tier, ≥50% of weight on leading dimensions. The floor is arithmetic, not taste — a dimension at 5% moves the composite five points across its whole range, less than one band width, so it cannot change a decision.

Sparse model (CRM + support + billing, no telemetry). Product usage prints as ❌ Missing at weight 0 and is not redistributed — silent redistribution manufactures a false green (Step 6). Expect AUC 0.62–0.72 [P], report deciles not probabilities, lean harder on override caps (which need no calibration to be useful), and carry a named instrumentation roadmap: login/MAU joined to account_id → 5–10 named value events → seat activation → entitlement vs use (references/model-archetypes.md §6).

Step 4 — Build a scoring function for each dimension

Raw values never enter a composite. Every dimension gets an explicit transform, a stated baseline and an NA rule. All formulas, half-lives, guards, U-curves and per-dimension recipes: references/scoring-functions.md — read it while writing §3 of the spec. 1. Score against the account's own baseline, not an absolute threshold — a weekly-cadence account that stays weekly is healthy; a daily account that drops to weekly is churning.

base   = median(weekly_value_events, weeks t-12 … t-5)     # median survives outages
recent = median(weekly_value_events, weeks t-3 … t)
trend  = (recent − base) / max(base, ε)
s_trend: ≥+0.10 → 100 | −0.10..+0.10 → 75 | −0.25..−0.10 → 50 | −0.50..−0.25 → 25 | <−0.50 → 0
Guard: require base ≥ 20 events, else return NA — not −100%.

2. Recency decays exponentially, one interpretable parameter: s_recency = 100 × exp(−ln 2 × days_since_touch / HL), HL = expected cadence in days, so one missed cadence scores exactly 50; half-lives per signal are in its §2. Never decay state — auto-renew being off does not become less true after 60 days.

3. Cohort-relative only where there is no natural scale, never for absolute reporting. Percentile rank within segment × tenure_band × product_tier, minimum cohort size 30 [P]. Cohort percentiles are zero-sum — half the book is below median by construction, so portfolio health can never improve.

4. Suppress noise before it fires (its §5): the consecutive-period rule — fire only after N=2 consecutive weekly periods below threshold, which cuts single-period false positives by roughly an order of magnitude [P]; seasonality masks over the customer's low season, holidays, academic summer and retail freeze windows; and a fleet-wide diff, because an event-taxonomy rename decays the whole book at once — if >15% of accounts move the same direction by >5 points in a day, suppress and page CS Ops.

Banded transforms for anything a human acts on, continuous for anything feeding a model or a ranking — mixing them undocumented is a top cause of "the score jumped 12 points and nobody knows why." Zero support tickets is not green: U-curve the support dimension (0 → 60, healthy band → 100, heavy volume → 20) and normalise per 100 licensed seats, or every large account is red.

Step 5 — Set the weights from outcomes, not from a workshop

Sample-size honesty. The classic rule is 10–20 churn events per predictor [Peduzzi et al., 1996 [A]] — 8 predictors means 80–160 events, not 80 accounts. Riley et al. (Statistics in Medicine, 2019) showed events-per-variable rules of thumb are unreliable and derived closed-form minimums accounting for prevalence and shrinkage [A]. Below the floor, ship v0 and say so. Cold start: ship the rubric, instrument everything, record score history from day one, and validate against proxy labels (downsell, escalations, sponsor departures, auto-renew disabled) until one renewal cycle closes. Then freeze weights for a full cycle before judging them — re-weighting monthly makes backtesting impossible, because every backtest is against a different score.

Step 6 — Add the non-compensatory layer

A weighted sum lets a 100 on Support offset a 0 on Sponsor, but real churn is usually one fatal condition, not an average of mediocrity. Three fixes, cheapest first: publish sub-scores as first-class (never render the composite without its 4–6 pillars); a worst-pillar rollup — the composite takes the band of its weakest pillar, or 0.7 × weighted_avg + 0.3 × min(subscores); and override caps. Defaults [DESIGN] — each is a decision or a structural fact, not an indicator, which is why an aggregate score must not wash it out (R2):

Non-renewal notice served or auto-renew switched off → cap ≤29, because that is the decision itself. Competitive evaluation or RFP confirmed → ≤49. Executive sponsor departed with no replacement named within 45 days → ≤49 — the contract has no owner on their side (R3). Invoice >60 days overdue → ≤49, because non-payment precedes non-renewal. CSM sentiment Red with written justification → ≤49, GitLab's exact production behaviour. Licence utilisation <50% at >180 days tenure → ≤59; partial churn has already happened. Data sufficiency under 70% of weight populated → suppress to "Insufficient Data", because an instrumentation failure is an ops defect, not a customer state. Each cap needs an evidence source and an expiry in the spec (assets/score-spec-template.md §4).

The NA rule is the most consequential line in the spec. The usual platform default redistributes a missing dimension's weight across the survivors in proportion to their existing weights — silently handing the biggest increase to whatever was already heaviest, a consequence GitLab documents in its own config. Choose explicitly between (a) proportional redistribution capped at 20% of original weight, (b) impute neutral, (c) composite = NA, (d) structural zero with the denominator printed (references/scoring-functions.md §6); and never let missing data produce a green. Staleness forces NA, because a frozen green on a dead feed is the worst failure in the catalogue — nothing about it looks wrong. GitLab's published policy: product usage → NA after 60 days without data; support → 30 days; CSM sentiment stale at 90 days, forced NA at 120 days — they "prefer to show nothing ('NA') over outdated data."

Step 7 — Segment the model

Segment when the relationship between a signal and the outcome differs, not merely when the level differs. If SMB and Enterprise both churn on usage decline at different absolute levels, you need different thresholds, not different models.

The five splits that earn their cost — lifecycle stage, touch model, entitlement tier, ACV band and pricing model — with the published GitLab and Benchmarkit anchors for each, are in references/model-archetypes.md §8. Read it before adding a variant. The two that are almost always right: NULL every trend measure for the first 30 days (onboarding accounts have no baseline), and mark relationship dimensions N/A, never zero, for tech-touch.

Prefer one model with segment-parameterised thresholds over five separate models. Ceiling: 3–5 variants [DESIGN] — each is a separate spec, backtest, band cut and governance review, so the fifth costs more than it discriminates.

Step 8 — Backtest against actual outcomes

Run scripts/backtest.py (worked example: scripts/sample-scores.csv). The eight-step protocol, the seven leakage traps, the full metric suite and the false-green / false-red taxonomy are in references/calibration.md — read §1 and §2 before the first snapshot, and §6 before the autopsy.

The four that decide whether the result means anything: snapshot point-in-time from the opt-out deadline, at T−180/120/90/60/30 · rebuild every feature as-of that date from immutable event logs, never current-state tables · split temporally, never randomly · exclude the leakage features — churn reason, renewal_status, cancellation-request ticket type, stage = Closed Lost, and CSM sentiment recorded after notice (GitLab's field refreshes every two hours, so it is contaminated within hours of a churn conversation). Then beat three baselines — constant base rate, single best feature, last quarter's model — compute per segment as well as overall, plot the lead-time curve, and autopsy 20 false-greens and 20 false-reds by hand, which produces more improvement per hour than any parameter search. Neslin et al. (JMR, 2006) found churn models suffered "very little decrease in performance" on a database compiled three months after the calibration data [A]; if yours collapses beyond T−45, suspect leakage in the short-horizon version.

The five metrics that decide it [DESIGN], in reporting order — full suite, definitions and targets in references/calibration.md §3: base rate in logos and ARR, computed per segment, because every other metric is read against it · top-decile lift, ≥2.5× minimum and 3–4× good — if CS only works the worst 10%, how much better than random? · ARR capture @ decile, ≥40% in the worst 10% and ≥65% in the worst 20%, because catching 80% of logos and 30% of dollars is a failure · PR-AUC at ≥2.5× the base rate, which is the metric to use instead of ROC-AUC whenever churn is under 10% · and the reliability curve, monotone and near-diagonal, the most persuasive chart a CCO will see and meaningful only once a map is fitted.

Accuracy is banned as a headline metric. At a 5% base rate, predicting nobody churns is 95% "accurate" and worth nothing. Turning the score into a probability, only after this backtest: hold out a validation set used for neither weights nor bands, bin into deciles, fit a monotone map — Platt under ~1,000 outcomes, isotonic above [P]. Re-fit the map quarterly, the weights far less often. Until it exists, publish bands and deciles, never a percentage (R22). Where .agents/cs-calibration.json exists, use its observed band rates and cite the sample size (../cs-context/scripts/calibrate.py).

Step 9 — Set thresholds from CSM capacity, not from F1

The red line is an operations decision (R13): you can only work the number of red accounts you have hours for, and a red list longer than capacity trains everyone to ignore all of it.

capacity      = CSMs × usable_hours_for_saves ÷ hours_per_save_motion    # usable ≈ 60% of nominal
red_threshold = the score where count(accounts below it) ≈ quarterly save capacity

Worked: 8 CSMs × 4 usable hours/week × 13 weeks ÷ 10 hours per save motion ≈ 41 save motions per quarter. That number sets the threshold — not F1, not Youden's J, not a round number. At a realistic operating point you accept roughly 38% precision and 48% recall, and that is the correct answer, not a compromise: working 400 accounts you do not have hours for saves less revenue than working 200 you do. Two floors [P]: below ~25% precision CSM credibility burns and alerts get ignored; below ~40% recall the score is not earning its build cost. scripts/backtest.py --capacity N prints the threshold table from your own data and fires both warnings (references/calibration.md §4 has the worked 2,000-account version).

Cut bands where the observed loss rate changes, not at round numbers. Published band cuts differ widely — GitLab's public config uses Green 75–100 / Yellow 50–74 / Red 0–49 — which is the point: no cut is canonical, because each is fitted to one book's distribution and one team's capacity. Derive yours, publish it, and never move it mid-year without restating history.

Step 10 — Ship explainability, or the score gets ignored

Every render must answer six questions with no analyst in the loop: what is the score · what was it 30 and 90 days ago · which three dimensions moved it most · which single dimension is furthest below its band · what is the recommended play · when was each input last refreshed.

Attribute to the delta, not the level. "Support is your lowest pillar" is true for half the book and tells nobody anything; "meeting recency fell 22 points in 30 days and drove 60% of the drop" is a work order. For an ML model, convert SHAP values into templated reason strings. Publish velocity beside level: a 62-and-falling account and a 62-and-rising account are different objects and one composite cannot express the difference. health_score_delta_30d is a first-class field, and green-but-falling ≥10 points in 30 days is the most valuable, most ignored cell in the model. Every band owns a play — action · owner · date · expected effect · success measure.

Step 11 — Govern it

One named owner in CS Ops; a score owned by a committee drifts, breaks quietly and dies. Semantic versioning: major = dimensions or label changed · minor = weights · patch = thresholds, and every version keeps its spec and its backtest permanently. Champion/challenger: a new config runs in shadow for one full renewal cycle before it drives a single play. Refresh: composite daily, history retained forever, calibration map and bands quarterly, weights semi-annually or after one cycle, redesign annually or on a pricing / product / segment change. Track score coverage (GitLab carried >95% of accounts as a company-level yearly goal) and band volatility. Publish the migration matrix before the rescore lands and tell each CSM which of their accounts move and why. A silent rescore destroys trust once, permanently.

Anti-gaming. Goodhart's law applies the moment the score is tied to comp, QBR review or CSM performance. The controls, one per gaming route, are in references/governance.md §4 — read it before any comp discussion. The two that matter most: mandatory written justification on CSM sentiment, with published per-CSM sentiment-vs-outcome calibration (a CSM whose greens churn at 3× the average is miscalibrated, not unlucky), and overrides capped at ≤10% of portfolio per quarter, each with an expiry. The tell that gaming is happening: score distribution improves while retention does not — plot them on one axis every quarter.

Never compensate on the health score itself. Compensate on retention and expansion outcomes and use the score to allocate attention; the moment the score is the target it stops being a measurement.

Output Template

Brief is the default. Emit this; offer the rest.

**<Company> health score — <verdict>. <the number that proves it>.**

<Two sentences naming the single largest defect or design decision, with its evidence and provenance tag.>

**Do:** <named action> — <owner> by <date>. Expected effect: <what moves, by how much>.
Confidence: <high / medium / low> (<X>/7 families). What would change it: <the falsifier>.

*Full specification, backtest and coverage ledger on request.*

Full emits the Specification, appends the Backtest Report when outcome data exists (references/calibration.md §8), and emits the Audit block in Audit mode (references/audit-checklist.md). Copy assets/score-spec-template.md verbatim for Full — never retype it from memory. It carries the Internal marking and direction line, the Bottom Line header table (predicts · population · archetype · weight method · validated? · red threshold · score coverage · as-of · confidence), all nine sections — §1 Prediction Sentence · §2 Dimensions with the hygiene line and rejected candidates · §3 Scoring Functions · §4 Non-Compensatory Layer · §5 Segmentation · §6 Bands and Plays · §7 Explainability Contract · §8 Remediation Roadmap · §9 Governance — and the Label & History Sufficiency table. Every artifact, Brief or Full, closes with these two blocks:

### Coverage Ledger
| Signal family | Source checked | Status | In score? | Weight | Oldest input | Notes |
|---|---|---|---|---|---|---|
| Product usage & adoption | | ✅/⚠️/❌ | | | | |
| Commercial & contract | | | | | | |
| Relationship & engagement | | | | | | |
| Support & reliability | | | | | | |
| Sentiment & VoC | | | | | | |
| Billing & payment | | | | | | |
| Firmographic & external | | | | | | |

**Coverage: X / 7 families (Y%) → confidence capped at <level>** (`R23`). Blind spots: <which families are missing,
and what they typically hide.> Anything absent elsewhere is `UNKNOWN — requires <source/field>`, never a benchmark.

### Assumptions
| # | Assumption | Why it was needed | If wrong |
|---|---|---|---|
| 1 | Label = non-renewal or downsell ≥15% of ARR | No label specified; `cs-context` §6 silent | A logo-only label drops every shrinkage event; ARR capture falls and the usage dimensions lose weight |
| 2 | Horizon 120 days to the opt-out deadline | Notice period present, target horizon not stated | At 90 days AUC rises and the save window shrinks below one motion; at 180 days expect more false reds |
| 3 | Red threshold at the worst decile | No capacity in `coverage-and-capacity` or `cs-context` §3 | If real capacity is half that, the red list is twice what the team can work and CSMs start ignoring all of it |

*(One row per default taken, each with a concrete consequence — "may affect results" is not one. Omit only when nothing was assumed, never silently.)*

Quality Bar

• The prediction sentence is written, with event, horizon, unit and population all named, the horizon measured to the opt-out deadline (R1) and labels dated at decision_date (R24)
• Every dimension maps to one of the seven signal families, passed the four-axis admission test, and carries an explicit transform, baseline, NA rule and staleness rule; rejected candidates listed with the axis they failed
• Weight hygiene verified — sums to 100, none <5%, none >35%, count 5–8, ≥50% leading — and the derivation method named with its sample size and limits
• Missing data can never produce a green: the data-sufficiency floor is stated and enforced, and override caps carry an evidence source and an expiry (R2)
• Segmentation justified by a relationship difference, not a level difference; backtest uses time-respecting splits
• Leakage features named and excluded; accuracy is not a headline metric, base rate is reported first, and capture is reported in ARR, not only logos
• Red threshold derived from stated CSM capacity with the arithmetic shown (R13)
• No probability stated without a fitted calibration map and its backtest cited (R22)
• Explainability contract answers all six questions, reason codes attribute to the delta, and every band has a play with action · owner · date · expected effect · success measure
• Coverage Ledger over all seven families, plus label & history sufficiency (R23)
• Gaps written UNKNOWN — requires X with no benchmark substituted for a company-specific figure, and every default taken appears in the Assumptions table with a concrete consequence
• Marked Internal; no health score, band or ARR-at-risk figure drafted for a customer (R18)
• The words "will churn", "guaranteed" and "100% accurate" do not appear
• Brief emitted by default, Full only on request; no competing CS platform named

Anti-Patterns

Related Skills

Going Deeper

…(truncated)