Ensure Prod

Ensure Main is on Prod

Verify that the current HEAD of main is deployed to the production k8s cluster. If it's not, diagnose why and take corrective action.

Step 1: Compare git HEAD with what's actually running on prod

# Get current HEAD
git rev-parse HEAD

IMPORTANT: You must verify against the actual running pods, not just Argo CD's sync status. Argo CD's REVISION field reflects the git manifest revision, NOT the container image — a "Synced" app can still be running old pods if the image build hasn't finished or the rollout hasn't completed.

Check what's actually running using both of these methods (run in parallel):

1. Public build-info endpoint (most reliable — shows exactly what the running pod reports): Use WebFetch on https://abaci.one/api/build-info and extract git.commit and git.commitShort. This also shows environment, git.isDirty, and build timestamp.

2. Pod image inspection via Kubernetes MCP: Use kubectl_describe on one of the abaci-app pods in the abaci namespace. Look at the Image field — the tag or digest tells you which build is running. Cross-reference with the image updater annotations if needed.

Match criteria: HEAD matches prod when the build-info endpoint's git.commit equals git rev-parse HEAD. If they don't match, proceed to Step 2.

Step 2: Check CI pipeline

# Recent workflow runs
gh run list --limit 5

# Check if the latest run on main succeeded
gh run list --branch main --limit 3

• If a build is in progress: Report "build in progress" with the run URL and estimated time (~13min for full build). Stop and let the user decide whether to wait.
• If the latest build succeeded: Argo CD image updater should pick it up within a few minutes. Check if the image updater has seen it (see Step 4).
• If the latest build failed: Go to Step 3.

Step 3: Check fixbot activity

# Open fixbot issues
gh issue list --label fixbot --state open

# Recent fixbot PRs
gh pr list --label fixbot --state open

• If a fixbot PR exists: Review it with gh pr view <number>. If checks pass and the fix looks correct, tell the user it's ready to merge. If it needs judgment, explain what needs review.
• If a fixbot issue exists but no PR yet: Fixbot is still working on it. Report the issue and wait.
• If no fixbot activity: Check if the diagnostic workflow ran: gh run list --workflow "Diagnose CI Failure" --limit 3. If it didn't, the failure may be too new or fixbot itself failed. Investigate the CI failure manually.

Step 4: Check Argo CD image updater

If CI passed but pods haven't rolled, check the image updater:

Use the Kubernetes MCP to:

1. Check kubectl logs -n argocd -l app.kubernetes.io/name=argocd-image-updater --tail=30 for recent activity
2. Check kubectl get applications -n argocd for sync status

If the image updater hasn't picked up the new image yet, it may just need a few more minutes. Report the status.

Step 5: Verify pod rollout

Once pods are rolling, monitor via the Kubernetes MCP:

1. kubectl get pods -n abaci -l app=abaci-app — check that pods are running the new revision
2. If pods are in CrashLoopBackOff or ImagePullBackOff, report the error

Step 6: Report

Provide a clear summary:

• Current HEAD SHA (short)
• Prod SHA (from build-info endpoint, short)
• Argo CD revision (short) — note if this differs from what pods are actually running
• CI status (passed/failed/in-progress)
• Fixbot status (if relevant)
• Pod status
• Any build metadata issues (e.g. environment showing "development" instead of "production", dirty flag set incorrectly)

Edge case: fixbot committed directly to main

If you discover fixbot committed directly to main (check git log --oneline -5 for commits by github-actions bot that aren't merge commits), push an empty commit to trigger a clean CI run:

git pull
git commit --allow-empty -m "ci: trigger build after fixbot commit <sha>"
git push

Then re-enter this procedure from Step 2.