Deployment Operations

Deployment Operations

Most of what belongs here is in the docs, because operators hit the same problems. Go there first; this file exists to route you and to record the habits that keep a diagnosis honest.

Read first

• o11y Troubleshooting — indexed by symptom, which is what you have when something is wrong. Error strings, pods that will not start, changes that appear to do nothing.
• Uninstalling — read before tearing anything down. Teardown deadlocks by default.
• Production Best Practices — the checklist, tagged with who owns each item.
• Installing via Helm and Terraform — the two paths.
• test/e2e/README.md — the local kind tiers.

Name the cluster, every time

kubectl and helm both take the current kubeconfig context when you do not say otherwise, and these operations install, restart, and delete. A context left pointing at production is the failure mode.

• The make e2e-* targets pin KIND_CONTEXT; the Terraform test substrate pins its providers. Do not add a command that inherits the ambient context.
• kubectl config current-context ignores --context — it reports the configured context, not the one a command used. Do not use it to prove which cluster you touched. Capture kubectl cluster-info instead; the server address is unambiguous.

Before diagnosing anything

A timeout is not a duration problem. "helm upgrade timed out" almost never means something was slow — assume something is broken, and raising the timeout is almost always the wrong response. Identical whether you ran helm or Terraform ran it for you. Inspect events first, then pod status, then the previous container's logs. See Troubleshooting.

Check your credentials before your work. AWS and GCP sessions commonly last 12 hours, so a morning login lapses mid-afternoon — and an expired session breaks everything at once, including things you did not touch. When a failure looks impossibly broad, that breadth is the tell. Re-authenticate, then debug.

Three habits that prevent wrong conclusions

Assert on recent data, not on any data. Loki's filesystem store survives a pod restart and WAL-replayed streams count toward streams_created_total, so an unbounded query and that counter both pass against a stack that stopped ingesting an hour ago. Bound the window, and keep it meaningfully shorter than how long a break would have lasted.

A green helm upgrade is not evidence the change took effect. Alloy's config arrives partly through envFrom, and environment variables are fixed at container start. Check the ConfigMap and whether the pod restarted.

Read the render output. The chart's validators print to NOTES.txt, and a warning there is usually the earliest signal. On the Terraform path they surface in the apply log — which is why the module disables subchart notes, so ours are not buried.

When you cannot reproduce it later

The cluster dies with the CI runner, and a red E2E with no artifacts costs more than the test saves. test/e2e/dump-diagnostics.sh collects pods, events, storage, describes for unhealthy pods, all container logs including --previous, the four Alloy ConfigMaps, and the cluster's identity. Run it before deleting anything, and extend it rather than gathering by hand.

Things that are the cloud, not us

Worth recognising quickly, because they look like misconfiguration:

• RESOURCE_POOL_EXHAUSTED / GCE out of resources — a capacity stockout. Narrow machine families (Arm, bundled local SSD) hit it more often.
• pd-balanced disk type cannot be used by <machine-type> — GCP C4/N4 take only Hyperdisk, and no default GKE StorageClass qualifies.
• Workload identity cannot be tested on kind. There is no OIDC issuer an IAM provider trusts, so IRSA, GKE Workload Identity, and Entra Workload ID are only exercised against real clouds. The chart validators assert the shape of that config instead, which is why a mismatched annotation fails at render time.

Storage changes are not free

volumeClaimTemplates are immutable, and Kubernetes does not garbage-collect PVCs created from them — persistentVolumeClaimRetentionPolicy defaults to Retain. So reinstalling does not reset a volume's class or size; the new StatefulSet binds the old PVC. Delete PVCs deliberately, and know what is on them before you do (see the per-workload notes in the troubleshooting page).