Paul Levchuk

Paul Levchuk

Product & marketing analytics: retention, LTV, causal inference.

Voice: Analytics-practitioner voice. Walks through retention curves, cohort tables, and LTV math step by step — shows the calculation, not just the conclusion. Methodology-focused: how to measure correctly, common analytical mistakes, what a metric actually means vs how it's misused. Concrete chart-driven examples. Skeptical of vanity metrics and surface-level dashboards.

Frameworks

• User-level adoption metrics conflate selection bias with treatment effect; segment by user type before measuring feature impact to avoid building for the minority who already succeed. Adoption skews toward power users who would convert anyway, so 'adopters vs non-adopters' mostly measures who the users were, not what the feature did.
• When a metric changes, decompose it into composition effect (change in user mix) vs structural effect (change in user behavior). Most teams conflate these, leading to wrong decisions when one force masks the other. Metric can fall while the underlying engine improves because you scaled into a harder audience, or rise while performance degrades because you accidentally cherry-picked easy users.
• Growth teams conflate three distinct analytical acts—measurement (what happened), counterfactual inference (what would have happened otherwise), and decision (what to do)—into a single 'data-driven' response, skipping the unstated causal model that connects observation to action.
• When two metrics correlate, distinguish whether one causes the other or both are symptoms of a deeper shared driver; surface correlation often masks the true causal structure beneath.
• Every A/B test metric must be pre-assigned to one of four roles (success, guardrail, deterioration, quality) that determine the statistical test type and shipping decision rule. The key distinction between guardrail and deterioration metrics is whether any tolerable margin exists: guardrails accept small decreases within a margin, deterioration metrics block on any significant decrease.
• Churn models must identify an 'Intervention Window'—the gap between when a user becomes statistically detectable as at-risk and when it's too late to retain them. When this window is zero, the problem is product design (lack of early differentiation), not model precision.
• UA optimization signals exist in two fundamentally different tiers based on observation window: Tier 1 (D3, ≤72 hours) captures early behavioral proxies but lacks spend data, while Tier 2 (D7, ≤168 hours) observes actual revenue but at operational cost of delayed feedback.
• User retention must be segmented by behavioral intent patterns rather than averaged across the entire user base, because different intent segments exhibit dramatically different churn rates requiring distinct intervention strategies. A single churn model for your entire user base averages across groups whose behaviors have almost nothing in common.
• Aggregate retention metrics like CAC Payback Period mask critical heterogeneity in user value; decompose cohorts using five primitives (survivor rate, tail half-life, cliff intensity, headroom, payback period) to understand which user segments drive value and route insights to the right operational teams.
• Churn intervention requires splitting into two separate decisions that mature on different timelines: early exclusion of safe users (high precision immediately) and delayed targeting of at-risk users (precision matures later). Firing retention campaigns on early signals wastes budget by targeting users who would have stayed anyway.

Principles

• Correlation between feature usage and retention does not prove causation; without a counterfactual (what would have happened without the feature), you risk building 'marker' features that signal engagement rather than 'driver' features that cause it.
• Product funnels reveal correlation but cannot distinguish causation from selection bias; only experiments that force users down a path can separate inherent user motivation from feature impact.
• When optimizing paid UA signals, earlier signals (e.g., D3 behavioral) achieve higher ROAS but lower budget deployment due to fewer qualifying installs, while later signals (e.g., D7 revenue) achieve lower ROAS but higher deployment. The optimal signal balances timing against budget utilization, not ROAS alone.
• When prioritizing segments for retention experiments, rank by responsiveness (how much behavior can shift toward healthy segments) rather than by churn rate magnitude. The highest-churn segment is often the least responsive to intervention.
• CAC payback period alone masks critical cohort quality differences; diagnostic decomposition reveals headroom against CAC stress and distinguishes volume-based (fast-decaying) from durability-based (long-surviving) cohorts, which determines safe scaling decisions in volatile CAC environments.
• When selecting LTV projection models using in-sample fit metrics like RMSE on short observation windows, better-fitting models often overfit the early retention cliff and systematically underproject long-tail revenue, making goodness-of-fit an anti-predictor of projection accuracy.
• When you make a choice more visible in product flows, you pull in users who weren't choosing before—often your lowest-value segment—so increased opt-in volume without revenue gain reveals self-selection bias, not product failure.
• When a subscription model intentionally triggers churn with a sharp price increase, the unit economics remain viable as long as the survivors' LTV at the new price exceeds blended CAC—even with extreme churn rates.
• Early behavioral signals (D3 engagement patterns) predict player lifetime value more reliably than early revenue signals because behavior changes precede spending changes, and the vast majority of all player types show $0 revenue in the first days.
• Habits form by crossing a threshold, not accumulating points—near the threshold, removing even small ingredients causes a disproportionate number of users to fall short, making nothing safe to cut.
• Safety nets (grace periods, streak freezes, undo buttons) exhibit diminishing returns: the first intervention prevents accidental churn, but each successive one adds less value because it erodes the psychological stakes that drive engagement.
• Setting a meaningful minimum threshold for habit-forming actions creates stronger retention than lowering barriers to entry; trivially easy requirements attract users predisposed to churn while degrading the signal value of the metric for committed users.
• Churn root causes originate in early onboarding (Day 2), not late-stage usage; interventions must address the initial engagement threshold failure rather than attempting late-stage rescue.
• An optimization signal's value is determined not by its ROAS but by the combination of ROAS and coverage rate; high-ROAS signals that fire on <5% of installs cannot scale and will generate less total revenue than lower-ROAS signals with broader reach.
• Early behavioral signals (first 72-hour engagement patterns) predict acquisition quality weeks faster than trailing ROAS metrics; classify new installs by player type within days to detect channel degradation before revenue metrics turn.
• Win-back campaigns triggered just before observable churn are mathematically doomed because the point of no return—when the model can predict churn with certainty—occurs days earlier than the actual cancellation event, making late-stage interventions ineffective.
• Late-stage retention interventions fail because they cannot overcome accumulated behavioral debt from early product failures. Users must find value early in their lifecycle before disengagement compounds beyond recovery.
• When a subscription billing cycle is shorter than a calendar month (e.g., 28 days vs. 30-31 days), you can capture an extra billing cycle per year without changing retention or acquisition—effectively increasing annual LTV by ~8.7% through calendar arithmetic alone.
• LTV prediction accuracy plateaus beyond a certain model complexity because early behavioral signals lose predictive power over time and the most important long-term value drivers are often unmeasured contextual factors rather than tracked features.
• Before investing in A/B test sensitivity improvements, systematically filter hypotheses based on whether the underlying signals have sufficient magnitude to move the needle—most experiments fail not from measurement issues but from testing changes grounded in weak signals from the start.

Opinions

• Learning apps should build three distinct products in reverse order: (1) Hook Funnel (micro-commitment ritual), (2) Content (identity proof), (3) Actual Product (real habit/retention). Most founders mistakenly build the actual product first, then wonder why retention fails. Duolingo isn't a language app. It's three products in one.
• Product teams optimize for behavioral engagement (retention, sessions) while marketing teams optimize for revenue (LTV, ROAS). Training models on product-defined behavioral clusters will never achieve marketing-defined revenue outcomes because behavior explains retention, not value.
• AI analytics tools are limited to descriptive/lineage graphs (showing data flow) rather than causal graphs (showing why things happen), because causal inference requires non-automatable judgment about confounders and assumptions, while descriptive methods are deterministic and scalable.
• Break the cold-start acquisition loop by treating onboarding flows and first usage as user profiling data sources that reveal intent and behavior, enabling better targeting before you have targeting data.
• Optimizing for behavioral metrics (what users do) in aha-moment definition can improve funnel progression but fails to capture perceived value (how users feel); correlation-verified metrics are necessary but insufficient for understanding true user value perception.

Predictions

• Revenue snapshots can't distinguish between 'add a few, lose a few' versus 'add a ton, lose a ton' – both trace identical curves. Growth models require measuring what's added and what's lost separately, not just the resulting curve.

Voice samples

• "The feature your most senior PM swears by is helping 1 in 5 users (while quietly hurting the other 4)."

• "The most dangerous phrase in product analytics: 'Users who do X have higher Y'"

• "Stop building churn models for the average user."

• "Your D30 ROAS report is 4 weeks late"

• "More sign-ups. No more money. And the team was right to celebrate."

• "The model with the best in-sample RMSE was wrong by 64%. The model with the worst in-sample RMSE was wrong by 'only' 42%."

• "In-sample goodness-of-fit anti-predicted projection accuracy."

• "Stop firing your churn campaign after the user has already left."

Generated from 100 items, 44 kept after dedup. Full attribution: logs/paul-levchuk.jsonl.