Staged Rollouts

Staged Rollouts

Instructions

Never ship a new mobile version to 100% of users at once. The point of a staged rollout is to catch the regressions your QA missed before they become a one-star-review avalanche or a rollback that you cannot execute (you cannot force users to downgrade).

1. Why Staged

• No instant rollback. A bad mobile build stays installed. Only new installs can receive a fix. Staging buys time.
• Heterogeneity. OS versions, device models, locales, carriers. Staged cohorts sample this variance.
• SDK surprises. New or updated SDKs can crash in production configurations you did not anticipate.
• Store review latency. A hotfix can take hours to days. Staged exposure is your primary safety net.

2. Canonical Percentage Schedule

Hold longer on the first step if the release includes new SDKs, major refactors, or schema migrations.

3. Halt Criteria

Write halt criteria before the release, not during incidents.

• Crash-free sessions drops more than 0.3 pp vs baseline -> halt.
• Crash-free users drops more than 0.2 pp vs baseline -> halt.
• ANR rate on Android doubles -> halt.
• Key funnel conversion (sign-in, checkout) drops more than 5% relative -> halt.
• Single crash hitting more than 0.1% of users -> halt until fixed.
• Store review delta below -0.3 stars within 6 hours of a step -> investigate before proceeding.

Halting is cheap. Ship the fix, resume the rollout from the halted step.

4. Feature Flags and Kill Switches

Staged rollout of the binary is coarse. Feature flags are fine-grained and instantaneous. Pair them.

• Every risky feature sits behind a server-side flag.
• Flags support: on/off global, percentage rollout, per-segment (country, platform, app version).
• Include a local fail-closed default: if the config service is unreachable, the flag evaluates to the safe value.
• Include a kill switch per feature: a flag that disables the feature regardless of other logic.

val flags = FeatureFlags.current()
val checkoutV2 = flags.boolean("checkout_v2", default = false)
val newPaymentFlow = flags.percentage("payment_flow_v2", default = 0.0)

Providers: LaunchDarkly, Firebase Remote Config, Unleash, self-hosted via a simple JSON endpoint on a CDN. Keep evaluation deterministic (hash userId) so a user stays in the same cohort across sessions.

5. Observability Hooks

Tag every metric and crash with the version and flag state.

• app.version, app.build, flag.<name> as dimensions.
• Dashboards show crash-free sessions per version and per flag.
• Alerts target the current release cohort, not global aggregates.

6. iOS Specifics

• App Store Phased Release is automatic once enabled, in daily 1/2/5/10/20/50/100 steps. You can pause and resume via App Store Connect.
• Pausing stops new users from getting the update; users who already updated keep it.
• Expedited reviews exist for serious issues and can cut review to hours.

7. Android Specifics

• Play staged rollout is configured per release track; you choose the percentage.
• Halt rollout stops new installs at the current version; users on it keep it.
• You can resume at the same or a new percentage. Bumping percentage works; lowering does not.
• Consider in-app updates (flexible or immediate) to nudge users toward the fix once shipped.

8. Data Migrations in Staged Rollouts

Migrations are the riskiest part of any rollout.

• Write migrations to be idempotent and forward-only where possible.
• Add dry-run and metrics to the migration itself (counts, timings, failures).
• Gate risky migrations behind a feature flag so an existing install can stay on the old schema until a follow-up release.
• Never require a server change synchronized to a client version; handle versions overlap explicitly.

9. Rollout Playbook

Write a brief playbook for each release:

# Release 2.5.0 Rollout

## Baselines (from 2.4.x)
- Crash-free sessions: 99.85%
- Crash-free users: 99.6%
- ANR rate: 0.09%

## Schedule
- iOS: Phased Release on, monitor daily.
- Android: 5% -> 20% -> 50% -> 100% with 24h holds.

## Halt criteria
- Crash-free sessions drop > 0.3pp
- Checkout conversion drop > 5%
- Any crash > 0.1% of users

## Kill switches
- flag: checkout_v2
- flag: new_sync_engine

## Owners
- Release manager: @alice
- Pager primary: @bob
- Pager secondary: @carol

## Rollback plan
- Halt immediately
- Ship hotfix branch from main (cherry-picked)
- Expedited review if iOS

10. Anti-Patterns

• "Staged rollout" that promotes every 2 hours. Too fast to see regressions.
• Halting, fixing, and resuming without a new baseline. Check the data.
• Feature flags without a kill switch for the whole feature.
• Long-lived flags. Remove after full rollout; they become a maintenance tax.
• No baseline. "Crash-free 99.7%" means nothing without the prior version number.

Checklist

• Baseline metrics from the prior version are recorded before the rollout.
• A written halt criteria document exists with absolute thresholds.
• Risky features are behind feature flags with kill switches.
• Feature flag evaluation is deterministic per user and fails safe.
• Metrics and crashes are tagged with version and flag state.
• iOS Phased Release is enabled; Android staged rollout percentages are set.
• Rollout playbook names owners and escalation path.
• Rollback plan covers hotfix and expedited review steps.
• Stale feature flags are removed after full rollout.