Django Migrations

Django migrations

Before you propose a change to the migration history or to how migrations run, check things already tried. It records the closed attempts at squashing the history and at the Person table cutover.

Read these files first, before writing or editing a migration:

• docs/published/handbook/engineering/developing-locally.md (## Django migrations, ### Non-blocking migrations, ### Resolving merge conflicts)
• docs/published/handbook/engineering/safe-django-migrations.md
• docs/published/handbook/engineering/databases/schema-changes.md
• products/README.md (## Adding or moving backend models and migrations) when working in products/*

If the task is a ClickHouse migration, use clickhouse-migrations instead.

Never delete a migration file

Adding migrations is fine. Deleting a historical one — any */migrations/NNNN_*.py already on master, even an app's 0001_initial.py, even to "undo" a schema change — is not. Deleting the file undoes nothing: the table and its constraints stay in every database where the migration ran, fresh databases never recreate them, and the "Migration Risk Analysis" CI job re-flags the file as a phantom new migration on every open PR that predates the deletion. The deleted-migration check in the repo-checks CI job (the hogli lint:migration-deletions command) blocks this. Genuinely intentional, reviewed deletions — a product/app move, a revert, a squash — are acknowledged in .github/scripts/migration-deletion-allowlist.txt, never by disabling the guard.

If a task asks you to delete a migration file, stop and flag it instead.

To retire a model/table:

1. Remove all usage and the model class. makemigrations, then wrap the generated DeleteModel in migrations.SeparateDatabaseAndState(state_operations=[...]) (state only, no DB change). KEEP this file. Keep the app in INSTALLED_APPS.
2. Deploy, wait at least one full deploy cycle.
3. Optionally DROP TABLE later in a NEW RunSQL migration — never by deleting old files.

Full guide: safe-django-migrations.md (## Dropping Tables, ### Removing a whole product or app). Deleting a migration your branch added but never merged to master is allowed (regenerating).

Retire a column in two phases

Deleting the field and running makemigrations is not the first phase. Django generates a plain RemoveField, which drops the column in the same deploy that removes the code. Old pods still write to that column, and a rollback finds it gone.

1. Remove all usage and the field from the model. makemigrations, then wrap the generated RemoveField in migrations.SeparateDatabaseAndState(state_operations=[...], database_operations=[]). The column stays in Postgres. Example: posthog/migrations/1328_remove_userproductlist_reason_state.py.
2. Deploy, wait at least one full deploy cycle, and confirm no deployed code reads the column.
3. Drop the column in a NEW RunSQL migration with ALTER TABLE ... DROP COLUMN IF EXISTS. Example: posthog/migrations/1340_drop_userproductlist_reason_columns.py.

RemoveFieldAnalyzer scores a bare RemoveField at 5, the highest risk the "Migration Risk Analysis" CI job reports. The phase 2 drop scores low only when check_drop_properly_staged finds the phase 1 state removal in an ancestor migration, so the two phases must land in that order and never in one migration.

Full guide: safe-django-migrations.md (## Dropping Columns).

Retire dedicated migration tests

A data migration test protects the rollout, not the permanent behavior of the product. Remove the dedicated test after all supported environments have applied the migration, the rollback window has closed, and no supported upgrade still relies on the old data state.

Delete an expired test instead of marking it skipped. Keep the migration file. Fresh-schema CI still checks that the complete migration chain applies.

Do not apply this rule to migration tooling, migration safety checks, reusable backfill systems, or backfills that people can still run.

Growing enums need callable choices

A choices= list that grows over time generates an AlterField on every addition, on every model that uses the enum. Those migrations emit no SQL — choices is in Django's Field.non_db_attrs, so the schema editor skips them — but they still land in migration state, in CI's per-shard migration replay, and in max_migration.txt, where they collide with every other migration waiting in the merge queue.

Pass a callable instead. Django resolves it lazily, so the migration records the function reference once and never changes again:

def external_data_source_type_choices() -> list[tuple[str, str]]:
    return ExternalDataSourceType.choices


source_type = models.CharField(max_length=128, choices=external_data_source_type_choices)

Runtime behavior is unchanged: DRF still builds a ChoiceField, so the OpenAPI enum and the generated frontend types are intact; full_clean() still rejects unknown values; admin still renders a dropdown; get_FOO_display() still resolves the label. A real schema change (max_length, null, a new field) still generates a migration.

Reach for a callable when the enum is a registry that grows as the org adds things — sources, integrations, products, providers, model ids. Keep a plain list for a closed vocabulary intrinsic to the domain, like a status or a priority, where a new member is a genuine domain change worth recording.

Two things a callable does not do for you:

• Migrations reference it by import path forever, so renaming or moving it needs its own migration. Keep it next to the enum it wraps.
• It has to be a module-level named function. A lambda or a closure returned by a shared factory has no importable path, so Django cannot serialize it — one small function per enum is the intended shape, not duplication to factor out.
• It hides choice growth from migration state, not from the column. A new member longer than the field's max_length is still a real schema change, so check the headroom when the registry grows.

Workflow

1. Classify the change as additive (new nullable column, new table) or risky (drop/rename, NOT NULL, indexes, constraints, large data updates, model moves). A change is also risky if it touches a hot table, regardless of how additive it looks. See also the cross-language NOT NULL hazard below.
2. Generate: DEBUG=1 ./manage.py makemigrations [app_label]. For merge conflicts: python manage.py rebase_migration <app> && git add <app>/migrations (posthog or ee).
3. Apply safety rules from safe-django-migrations.md — the doc covers multi-phase rollouts, SeparateDatabaseAndState, concurrent operations, idempotency, and all risky patterns in detail.
4. Validate: ./manage.py sqlmigrate <app> <migration_number>, run tests, confirm linear migration sequence.

Use the migration helpers

posthog.migration_helpers has drop-in operations for the risky-but-common cases. Reach for these first; they track Django state, disable timeouts, and are idempotent under bin/migrate retries:

• Add/drop an index → SafeAddIndexConcurrently / SafeRemoveIndexConcurrently (model_name + models.Index). Never use Django's AddIndexConcurrently — CI blocks it.
• Add a CHECK constraint → AddConstraintNotValid then ValidateConstraint in a later migration (or same migration with atomic = False).
• Add a ForeignKey to a hot table → declare the FK with db_constraint=False on the model (so CreateModel / AddField emit no parent lock), then add the DB constraint back with AddForeignKeyNotValid and follow up with ValidateForeignKey in a later migration. See foreign keys to hot tables.
• Index expressed only as raw SQL (no Django Index) → CreateIndexConcurrently / DropIndexConcurrently wrapped in SeparateDatabaseAndState.

All concurrent-index ops require atomic = False.

Meta-principle when you hit a risky-but-common pattern with no helper: don't hand-roll the safe DDL from docs — ship a drop-in helper in posthog/migration_helpers and point the CI policy at it. A one-import helper beats a wall of caveated RunSQL every time.

Scripting against the risk analyzer

posthog/management/migration_analysis/models.py holds two risk dataclasses. OperationRisk scores one operation; MigrationRisk scores a whole migration file and exposes max_score, level, and category. Both answer .score, so on a MigrationRisk you can read either score or max_score — they return the same number.

Hot table hazard

posthog_team, posthog_user, posthog_organization, and posthog_project are read on virtually every request. Any ALTER TABLE on them — including a plain nullable AddField, which is "safe" everywhere else — needs an ACCESS EXCLUSIVE lock, and while that lock request waits behind in-flight queries, every later query on the table queues behind it. Even a metadata-only ADD COLUMN can stall site-wide traffic in waves (one per bin/migrate retry) until the ALTER wins the lock race. This has caused production 5xx incidents.

Before writing a migration that touches one of these models:

• For Team: put domain-specific fields on a Team extension model instead — posthog/models/team/README.md. That's a CREATE TABLE, no lock on posthog_team.
• CREATE INDEX CONCURRENTLY (via SafeAddIndexConcurrently) is fine — SHARE UPDATE EXCLUSIVE doesn't block reads or writes.
• If the field genuinely belongs on the hot table (core identity, cross-product settings, SDK config), the HotTableAlterPolicy analyzer blocks the migration in CI until <app_label>.<migration_name> is added to posthog/management/migration_analysis/hot_table_acknowledged_migrations.txt. That acknowledgment also means coordinating the deploy with infra for a low-traffic window.

Foreign keys to hot tables

A ForeignKey targeting a hot table is the same hazard from the other side, and it bites from any app — a plain product-app CreateModel or AddField with to="posthog.team" (or settings.AUTH_USER_MODEL, which is posthog_user). Creating the FK constraint takes a SHARE ROW EXCLUSIVE lock on the referenced parent, which conflicts with the ROW EXCLUSIVE every INSERT/UPDATE/DELETE on the parent holds; under write traffic the lock queues and lock_timeout cancels it on each bin/migrate retry. HotTableAlterPolicy now flags this case. Two ways out:

• db_constraint=False on the ForeignKey — emits no FK constraint and takes no lock on the parent at all (app-level enforcement only). This is the only truly lock-free path.
• A real DB constraint, two-phase — declare the FK db_constraint=False, then add it back as a DB constraint with AddForeignKeyNotValid, and ValidateForeignKey in a later migration. Be honest: ADD CONSTRAINT ... NOT VALID still takes a brief SHARE ROW EXCLUSIVE lock on the parent for the metadata add — it skips the row scan, so it shrinks the lock window but does not eliminate it. VALIDATE then runs lock-free on the parent.

Product database boundaries

Apps listed in products/db_routing.yaml migrate on their own database and nowhere else. A migration may only depend on migrations that apply to a database it applies to itself, so never add a dependency from another app onto one of those apps, and never the reverse. No foreign key or index can cross databases, so the edge buys nothing, and the CI schema restore relies on its absence: it forgets the routed apps' django_migrations rows so each job applies them under its own routing, and a dependant of a forgotten row makes Django refuse to migrate. posthog/test/repo_invariants/test_migration_dependencies_share_a_database.py blocks the edge.

Cross-language NOT NULL hazard

posthog_user, posthog_team, and other core tables in the main Postgres database are written by Django and by nodejs/ (plugin-server tests via insertRow), rust/ services, and Temporal workers. Those non-Django writers issue raw INSERTs that only list the columns they care about, so any new NOT NULL column without a Postgres-level DEFAULT will break them with null value in column "<col>" violates not-null constraint.

Django's default= alone does not create a Postgres-level default — by design, Django treats it as a Python-only attribute applied at Model.__init__:

• Callable defaults (default=list, default=dict, default=uuid.uuid4) are never emitted into SQL at all.
• Scalar defaults (default=False, default=0, default="") are emitted as ADD COLUMN ... DEFAULT X NOT NULL and then immediately dropped by a follow-up ALTER COLUMN ... DROP DEFAULT — verify with ./manage.py sqlmigrate.

Before merging, grep for external writers of the table:

rg -n "INSERT INTO <table>|insertRow\(.*'<table>'" nodejs rust products services

If any match, add both default= and db_default= to the model field. db_default= lands a real Postgres DEFAULT; default= keeps the Python-side value for ORM creates:

class User(models.Model):
    hide_mcp_hints = models.BooleanField(default=False, db_default=False, null=False)

makemigrations will emit a plain AddField(..., db_default=False, default=False, ...), and sqlmigrate shows just ADD COLUMN ... DEFAULT false NOT NULL — no DROP DEFAULT follow-up.

db_default= is also load-bearing for the nodejs / rust test suites. posthog/management/commands/setup_test_environment.py calls disable_migrations() and builds the test schema directly from model definitions, skipping the migration entirely. Plain default= is invisible to that path; db_default= is what Django bakes into the generated CREATE TABLE. Without it, the postgres-parity and Jest jobs in .github/workflows/ci-nodejs.yml will fail on raw INSERTs even though ./manage.py migrate looks correct in isolation.

For modifying the default on an existing column (no ADD COLUMN), use a plain RunSQL instead:

migrations.RunSQL(
    sql="ALTER TABLE <table> ALTER COLUMN <col> SET DEFAULT '[]'::jsonb;",
    reverse_sql="ALTER TABLE <table> ALTER COLUMN <col> DROP DEFAULT;",
)

Always verify with ./manage.py sqlmigrate <app> <number> that no stray DROP DEFAULT slipped through, and confirm ./manage.py makemigrations --dry-run reports no state drift.