Gitlab Best Practices

gitlab-best-practices

Run and upgrade a self-managed GitLab on the official Helm chart without unplanned downtime. The hard part is never helm upgrade. It is that the chart changes defaults the site never set, ships prerequisites written for a configuration the site does not run, and removed its own bundled databases in a release that breaks the tooling before it breaks the deployment.

Facts here were verified 2026-08-29 against machine-readable upstream sources, the unpacked chart, and a live install taken through a multi-hop 18.x → 19.x campaign. Re-verify anything version-bearing.

Version anchor (2026-08-29): latest chart 10.3.1 / app 19.3.1. Chart major = app major − 9. Required stops through 19.x: 18.2 · 18.5 · 18.8 · 18.11 · 19.2 · 19.5 · 19.8 · 19.11 — enumerate them fresh, never from memory.

Symptom index

The ladder

Stops are minor-level, and the .0 of a new major is not one. Read them from the machine-readable source, never from prose docs or memory:

curl -sS https://gitlab.com/gitlab-org/gitlab/-/raw/master/config/upgrade_path.yml
helm search repo gitlab/gitlab --versions | head -30   # chart <-> app, unfiltered

There is no 19.0 entry. The path is 18.11 → 19.2, which crosses every 19.0 and 19.1 breaking change in a single hop with no intermediate resting point. Satisfy 19.0's prerequisites before leaving 18.11.

A stop is a minor, satisfied by any patch of it — take the highest. Chart and app patch numbers drift apart (chart 9.11.12 carries app 18.11.11); name both in every artifact.

The nine facts that cause unplanned downtime

1. PostgreSQL 17 is both the minimum and the maximum for 19.x. There is no running ahead to PG 18. Because 18.x already tolerates PG 17, that overlap is the only ordering with a supported resting point on both sides: move the database to 17 while still on 18.x, then upgrade GitLab. Extensions are the quiet half — amcheck became required at 18.4, migrations do not install extensions, and the extensions docs page does not carry the table. → references/external-deps.md

2. Chart 10.0 makes external PostgreSQL, Redis and object storage mandatory — and breaks the tooling first. The bundled databases are gone and NOTES.txt carries hard fails, so any "render the chart with defaults" step dies, taking the all-features image sweep with it. Installs already on external dependencies are insulated at the deployment layer and still lose the diffing layer. → references/upgrade-campaign.md

3. Default flips are invisible to a values diff. A site inheriting a default cannot see it change by looking at what it sets. Crossing chart 9.11.12 → 10.2.5 flipped five defaults; a hand-built breaking-change list built from the release notes missed one. Diff stock values every hop, including against a hand-built list. Left unpinned, global.ingress.enabled: false alone takes the instance off the network.

4. A prerequisite in release notes is written for the DEFAULT configuration. Chart 10.3.0's "apply the Gateway API CRDs before upgrading or it fails on the GitLab Shell TCPRoute" reads as mandatory for everyone. It fires only where Gateway API objects actually render. Render the target with the real values, grep for the object kind the prerequisite names, and if it does not render, record why it cannot apply. Complying anyway is not free — Gateway API CRDs are cluster-scoped and permanent.

5. Offline helm template corrupts exactly the artifacts the diff gates read. A cluster-less render cannot populate .Capabilities.APIVersions, so the chart falls to its oldest apiVersion branch and silently drops blocks. helm upgrade is unaffected because it talks to the cluster — so only the diffs are wrong. Pass -a/--api-versions from kubectl api-versions; --kube-version does not fix it.

6. A single Sidekiq replica removes a silent migration-loss race. With multiple replicas, an old-version pod can take a background-migration job before terminating, crash, and leave an orphaned dedup key in Redis — after which migrations never run while the install looks completely green. Scale Sidekiq to 0 for the upgrade, or run one replica. Note the tension: the chart's zero-downtime procedure requires multiple replicas. → references/failure-modes.md

7. There is no GA high-availability path for Git data on Kubernetes. Standalone Gitaly went GA at 18.11 and is by design a single point of failure; Gitaly Cluster (Praefect) on Kubernetes is beta, and its first-class-solution epic publicly reports "No progress" at ~5 contributor hours a week. The stateless tier is genuinely HA-capable. Plan around a restore plan for Git data, not replication. → references/ha-and-topology.md

8. Rollback is a restore plan. Reverting means reverting the database schema, which needs a backup taken at the exact version and edition being downgraded to, and the restore overwrites all newer content. There is no supported helm rollback after migrations have run against production data — the documented remedy for a bad state is to go forward to a required stop. helm rollback recovers a bad values change, nothing more.

9. Do not tune the toolbox backup — split it by data class. Every blob round-trips object storage → pod disk → gzip → tar → back on every run, and the three obvious levers are all unavailable there: COMPRESS_CMD never reaches the blob path (gzip is hardcoded), SKIP=tar is refused with object storage, and backup-utility has no incremental mode at all. Move PostgreSQL to its own tooling, blobs to bucket-to-bucket replication, and repositories to Gitaly server-side backups. Upstream's documented cron.extraArgs recipe is defective — it pairs --skip repositories with --repositories-server-side, and skip wins, so the scheduled backup contains no Git repository data and nothing says so until restore. Drop --skip repositories. → references/backup-restore.md

AI, in three lines

• Free tier gets no Duo at all, and CE can never hold a licence, so no Duo path exists there ever. The chart defaults to edition: ee — which is what keeps an unlicensed install upgradeable later.
• Duo against a self-hosted vLLM needs Premium/Ultimate plus the Duo Enterprise add-on. Owning the GPUs does not buy out the licence. The mandatory AI Gateway defaults to calling customers.gitlab.com, and when that is unreachable and unoverridden it costs 20 seconds per request — presenting as "the AI is slow", not as an egress error.
• The MCP server is free (Tier: Free, Premium, Ultimate) and was decoupled from Duo in 19.2 — no add-on, no AI Gateway, no egress. On ≥19.2 that is the zero-licence answer; below it, the REST/GraphQL API.

→ references/duo-ai.md

Working method

• Run all six diff layers, every hop. Stock values, stock render, site values, set-difference verify, site render, and site-render-vs-live-cluster. Each catches what the others structurally cannot — the stock-render layer is the only one that sees RBAC, probes and container args, because those never appear in values.yaml. → references/upgrade-campaign.md
• Never patch a values delta onto a new version; use diff3. patch has two inputs and cannot know what the file looked like when it was edited — it has already fuzz-matched a hunk onto an adjacent identically-shaped key and silently changed cluster-wide behaviour, exit 0. Merge mechanics live in the helm skill (its own references/values-porting.md), not here.
• helm lint is not a typo check, and never kubectl apply a rendered template — it rotates lookup-generated TLS/CA secrets.
• Use glab, not a scraper, for gitlab.com. The notes endpoint 401s to unauthenticated fetchers and comments are JS-rendered. A scraped pass once concluded a fix had no backport; glab api .../related_merge_requests returned three and proved the opposite. A milestone says when a fix landed on master, not whether it reached the target version — only backport MRs and their target_branch answer that.
• Get background migrations to zero at every stop, not just the last. Active is fine; paused or failed is not.
• A failed helm upgrade is usually a timeout on a long migration, not a broken release. Check before reacting.
• Verify in order of what each check proves, cheapest first: helm list → Ingress objects exist with an external address → HTTP 200 on the web host → registry /v2/ returns 401 → background migrations clean.
• Citations, per-claim, with issue states: references/sources.md.