Deploy Status

Deploy Status

Monitor the real civitai PROD deploy chain — Tekton build -> Flux image automation -> Flagger canary -> prod primaries — by reading live cluster state with kubectl. The GitHub Deployments API is retained as a public cross-check (its source is the github-create-deploy Tekton task).

Primary truth = cluster state (this skill's status / watch). Cross-check = GitHub Deployments (gh-* commands below).

Cluster requirement (READ-ONLY)

All status / watch queries run against the DataPacket cluster via kubectl context civit-datapacket. One kubeconfig covers the whole chain (build ns tekton-builds, Flux ns flux-system, app+canary ns civitai-dp-prod).

This skill performs status reads only — kubectl get / kubectl logs. It NEVER mutates the cluster (no apply/delete/patch/rollout/scale). To trigger or roll back a deploy, use the talos-infra dp-build-deploy skill instead.

CRITICAL — prod is NOT keyed off the semver tag

A release maps to prod via the release-branch commit, not the tag object. Getting this wrong reports a stage/next build as prod.

The semver tag (vX.Y.Z) fires a different trigger (civitai-app-tag-trigger) that builds civitai-web (stage) on the tag commit. The PROD run is the civitai-web-build-* on the release-branch commit. Example: tag v5.0.1817 -> tag run on commit 5f19831 (civitai-web, NOT prod); the prod run is release commit 3dcd1e8 = civitai-web-build-cp9kz.

The prod selector that enforces this:

kubectl --context civit-datapacket -n tekton-builds get pipelinerun \
  -l pipeline=build-and-push,pipeline.jquad.rocks/git.repository.branch.name=release \
  --sort-by=.metadata.creationTimestamp

Usage (cluster — primary)

Driver script: deploy-chain.sh (in this skill dir). Dependency-light bash + kubectl.

• bash deploy-chain.sh status — full chain snapshot for the latest release-branch prod run.
• bash deploy-chain.sh status <sha> — snapshot for the prod run matching that commit (full or short sha, matched on the git.repository.branch.commit label). A raw sha for a tag/stage build correctly resolves to no prod run (it is never reported as prod).
• bash deploy-chain.sh status <tag> — a semver tag (e.g. v5.0.1817 / 5.0.1817, must contain a dot) is resolved to the release-branch HEAD via gh, then matched to its prod run.
• bash deploy-chain.sh watch [<tag|sha>] — poll the chain to completion, printing each transition; exits 0 only when every serving pool is fully ROLLED onto the new image (updatedReplicas == readyReplicas == desired for all app deployments), exits 1 if the build fails or a canary rolls back. Not merely when the deploy spec image flips.

What status prints (the full phase chain)

1. BUILD — PipelineRun .status.conditions + per-task TaskRuns (ns tekton-builds). Task order: notify-preparing -> github-create-deploy -> fetch-repository -> build-image -> migrations (plus trailing github-*/notify-* tasks). github-create-deploy is what writes the GitHub Deployments entries.
2. IMAGE PICKED UP — Flux ImagePolicy flux-system/civitai-prod-release .status.latestImage (ghcr.io/civitai/civitai-prod:<14-digit-ts>-<sha7>). ~1m GHCR scan.
3. TAG PINNED TO GIT — ImageUpdateAutomation (flux-system/civitai-dp-prod) commits the new tag to trunk (<=5m); reflected once primaries pick it up.
4. CANARY — two Flagger Canary CRs in civitai-dp-prod: civitai-dp-prod (SSR) and civitai-dp-prod-api (API / tRPC path). Reads .status.phase, .status.canaryWeight, .status.iterations, .status.failedChecks + recent Warning events (rollbacks).
5. PRIMARIES (100% prod) — civitai-dp-prod-primary (SSR), civitai-dp-prod-api-primary (API), plus the other serving pools (-api-heavy, -jobs, …). Deploy is fully live ONLY when every app deployment is fully ROLLED onto the new image — updatedReplicas == readyReplicas == desired — NOT when the deploy spec image flips. Flagger promotes the spec image minutes before the pods finish their rolling update, so a procedure added in the new image is NOT_FOUND on the not-yet-rolled pods. Never tell anyone a change is live off the spec image / status's image line alone — wait for the "FULLY ON PROD — every app deployment rolled" summary (or watch exit 0). The API primary fully rolled is what makes tRPC procedures live; the jobs pool fully rolled is what makes cron changes live.

Then an overall SUMMARY line: where in the chain + ETA (building / awaiting Flux pickup / canary progressing / rolled back / fully on prod).

Canary progression: Initialized -> Progressing (10->50% in 2-min steps) -> Promoting -> Finalising -> Succeeded. At rest between deploys a canary sits at Succeeded weight=0. End-to-end ~35-40 min (build 15-25).

Manual one-liners (same queries the script runs)

CTX="--context civit-datapacket"
# Latest prod run name
kubectl $CTX -n tekton-builds get pipelinerun \
  -l pipeline=build-and-push,pipeline.jquad.rocks/git.repository.branch.name=release \
  --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}'; echo
# Resolve a specific commit -> prod run
kubectl $CTX -n tekton-builds get pipelinerun \
  -l pipeline=build-and-push,pipeline.jquad.rocks/git.repository.branch.name=release,pipeline.jquad.rocks/git.repository.branch.commit=<FULL_SHA> \
  -o jsonpath='{.items[-1].metadata.name}'; echo
# Build condition + tasks
kubectl $CTX -n tekton-builds get pipelinerun <RUN> -o jsonpath='{.status.conditions[0].reason}: {.status.conditions[0].message}'; echo
kubectl $CTX -n tekton-builds get taskrun -l tekton.dev/pipelineRun=<RUN> --no-headers
# Flux image
kubectl $CTX -n flux-system get imagepolicy civitai-prod-release -o jsonpath='{.status.latestImage}'; echo
# Canaries (both)
kubectl $CTX -n civitai-dp-prod get canary
kubectl $CTX -n civitai-dp-prod get canary civitai-dp-prod-api -o jsonpath='phase={.status.phase} weight={.status.canaryWeight} failedChecks={.status.failedChecks}'; echo
# Primaries
kubectl $CTX -n civitai-dp-prod get deploy civitai-dp-prod-primary     -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
kubectl $CTX -n civitai-dp-prod get deploy civitai-dp-prod-api-primary -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
# Canary rollback / warning events
kubectl $CTX -n civitai-dp-prod get events --field-selector type=Warning --sort-by=.lastTimestamp | tail -10

GitHub Deployments cross-check (public, no cluster)

Useful when you can't reach the cluster or want to confirm the github-create-deploy task's view. This is the secondary source now.

The environment name is not prod — it is cluster-prefixed, and which prefix you see depends on when you look: a run is created as dp-prod and settles to do-prod, so the same deployment id reports different values minutes apart (original_environment keeps the first). Match on the suffix. An == "prod" filter returns an empty list for every deployment this repo has ever created, and an empty list reads as "no prod deploys found" rather than as a broken filter.

# Latest prod deployment + state
DEPLOY_ID=$(gh api repos/civitai/civitai/deployments --jq '[.[] | select(.environment | endswith("prod"))] | .[0].id')
gh api "repos/civitai/civitai/deployments/$DEPLOY_ID" --jq '{sha: (.sha[:7]), env: .environment, created: .created_at}'
gh api "repos/civitai/civitai/deployments/$DEPLOY_ID/statuses" --jq '.[0] | {state: .state, description: .description, updated: .updated_at}'

# Recent prod deployments with current state
gh api repos/civitai/civitai/deployments --jq '[.[] | select(.environment | endswith("prod"))] | .[:5] | .[].id' | while read id; do
  INFO=$(gh api "repos/civitai/civitai/deployments/$id" --jq '"\(.sha[:7]) \(.created_at)"')
  STATUS=$(gh api "repos/civitai/civitai/deployments/$id/statuses" --jq '.[0] | "\(.state)\t\(.description)"' 2>/dev/null || echo "unknown")
  echo "$id  $INFO  $STATUS"
done

# Specific commit
gh api "repos/civitai/civitai/deployments?sha=COMMIT_SHA" --jq '.[] | {id: .id, env: .environment, created: .created_at}'

GitHub Deployment state mapping: queued = build, in_progress = rolling out, success = done, failure = failed, inactive = superseded.

Build logs (Tekton)

Cluster (preferred):

RUN=<civitai-web-build-xxxxx>
# build-image pod logs (BuildKit output / compile errors)
POD=$(kubectl --context civit-datapacket -n tekton-builds get taskrun \
  -l tekton.dev/pipelineRun=$RUN,tekton.dev/pipelineTask=build-image \
  -o jsonpath='{.items[0].status.podName}')
kubectl --context civit-datapacket -n tekton-builds logs $POD --container step-build-and-push --tail=120

Dashboard (browser, GitHub-org auth) for a run: https://tekton.civitai.com/#/namespaces/tekton-builds/pipelineruns/<RUN_NAME>.

Debugging cheatsheet

• Build stuck/slow: check build-image TaskRun + pod logs (above). Common: OOM on Next.js build, BuildKit lock/disk on worker-spare.
• Image not picked up: Flux ImagePolicy latestImage not advancing — tag must match ^\d{14}-[a-f0-9]+$. Check ImageRepository civitai-prod-release scan.
• Canary rollback: get events --field-selector type=Warning on the prod namespace. Flagger rolls back when the success-rate / latency checks fail repeatedly; the configured thresholds live in the Canary CR, not here. Check .status.failedChecks on both canaries.
• Prod stuck on old image: confirm both primaries' images; if policy has the new tag but primaries don't after >10m, the ImageUpdateAutomation commit or Kustomization reconcile is lagging — escalate to talos-infra (dp-build-deploy skill).

Access levels