Recover from a stale state lock
A crash or killed CI job leaves a lock with no process behind it. Force-unlocking a lock that is still held corrupts state; confirm the holder is gone first.
Procedure
- Read who holds the lock; the error names the ID, path, and operation: Error: Error acquiring the state lock ID: 8f3c... Operation: OperationTypeApply Who: ci-runner@...
- For S3+DynamoDB, inspect the lock row:
aws dynamodb get-item --table-name tf-locks
--key '{"LockID":{"S":"acme-tfstate/prod/terraform.tfstate"}}' - Confirm no live process holds it: the CI job that owns the ID is finished or failed, and no
terraform applyruns on that runner. ps aux | grep '[t]erraform apply' - Only then force-unlock with the exact ID from the error: terraform force-unlock 8f3c-...
- If a run genuinely is in progress, wait for it;
-lock-timeout=5mmakes waiting automatic. - After force-unlock, immediately
terraform planto confirm state is intact and consistent. - Do not delete the DynamoDB row by hand instead of force-unlock; the CLI also notifies the backend and keeps lock metadata consistent.
- Fix the root cause: set
-lock-timeout, add CI concurrency limits, and ensure killed jobs do not orphan locks.
Pitfalls
- Force-unlocking a lock held by a live apply, so two writers interleave and corrupt state.
- Deleting the lock table row manually, leaving the backend's lock metadata inconsistent.
-lock=falseas a "fix", removing locking for every future run.- Force-unlocking with a guessed ID; the CLI requires the real one, and people work around it by deleting the row.
- Not running a plan after unlock, so pre-existing corruption surfaces at the next apply.
Verification
aws dynamodb get-item --table-name tf-locks \
--key '{"LockID":{"S":"acme-tfstate/prod/terraform.tfstate"}}' # Item: null after unlock
terraform plan # acquires lock, no errors
Report: the stale lock ID, the evidence the holder was dead, the force-unlock result, and the clean plan afterward.