Crowdsec Agent Registration Failure

CrowdSec Agent Registration Failure

Problem

After a CrowdSec LAPI restart or redeployment, agent DaemonSet pods lose their credentials but LAPI retains the old machine registrations. When agents try to re-register with the same pod name, the wait-for-lapi-and-register init container fails with user already exist, causing CrashLoopBackOff with hundreds of restarts.

Context / Trigger Conditions

• Agent init container logs show: Error: cscli lapi register: api client register: api register ... user 'crowdsec-agent-xxxxx': user already exist
• Agent pods show status CrashLoopBackOff or Init:CrashLoopBackOff with many restarts
• kubectl describe pod shows BackOff restarting failed container wait-for-lapi-and-register
• LAPI pods were recently restarted or redeployed
• cscli machines list on LAPI shows entries matching the stuck agent pod names

Solution

Step 1: Identify stuck agents

kubectl --kubeconfig $(pwd)/config get pods -n crowdsec

Note the pod names that are in CrashLoopBackOff (e.g., crowdsec-agent-jr5q7).

Step 2: Confirm the init container error

kubectl --kubeconfig $(pwd)/config logs -n crowdsec <agent-pod> -c wait-for-lapi-and-register --tail=5

Should show user already exist error.

Step 3: Find a running LAPI pod

kubectl --kubeconfig $(pwd)/config get pods -n crowdsec | grep lapi

Step 4: Delete stale machine registrations from LAPI

kubectl --kubeconfig $(pwd)/config exec -n crowdsec <lapi-pod> -- cscli machines delete <agent-pod-name>

Repeat for each stuck agent.

Step 5: Wait for agents to recover

The agents are in CrashLoopBackOff with exponential backoff (up to 5 minutes). They'll automatically retry registration and succeed after the stale entry is deleted. This can take up to 5 minutes per agent depending on where they are in the backoff cycle.

Verification

# All agents should show Running status
kubectl --kubeconfig $(pwd)/config get pods -n crowdsec | grep agent
# DaemonSet should show all pods READY
kubectl --kubeconfig $(pwd)/config get ds -n crowdsec

Example

# Identify stuck agents
$ kubectl get pods -n crowdsec | grep agent
crowdsec-agent-jr5q7  0/1  CrashLoopBackOff  485  3d
crowdsec-agent-jw76q  1/1  Running            8    3d
crowdsec-agent-mtgxh  0/1  CrashLoopBackOff  483  3d
crowdsec-agent-pfw2l  0/1  CrashLoopBackOff  481  3d

# Delete stale registrations
$ kubectl exec -n crowdsec crowdsec-lapi-xxx -- cscli machines delete crowdsec-agent-jr5q7
level=info msg="machine 'crowdsec-agent-jr5q7' deleted successfully"
$ kubectl exec -n crowdsec crowdsec-lapi-xxx -- cscli machines delete crowdsec-agent-mtgxh
$ kubectl exec -n crowdsec crowdsec-lapi-xxx -- cscli machines delete crowdsec-agent-pfw2l

# Wait ~5 minutes, then verify
$ kubectl get pods -n crowdsec | grep agent
crowdsec-agent-jr5q7  1/1  Running  1  3d
crowdsec-agent-jw76q  1/1  Running  8  3d
crowdsec-agent-mtgxh  1/1  Running  1  3d
crowdsec-agent-pfw2l  1/1  Running  1  3d

Notes

• This is a known limitation of the CrowdSec Helm chart — the init container registration script is not idempotent (it doesn't handle "already exists" by deleting and re-registering).
• The cscli machines list output will show many historical stale entries from past DaemonSet rollouts. These are harmless but can be cleaned up if desired.
• This issue also causes the CrowdSec blocklist import CronJob to fail, since it selects agent pods alphabetically and may pick a non-running one. Fixing the agents also fixes the blocklist import.
• See also: k8s-nfs-mount-troubleshooting for other common pod startup failures.