# CI CD Quirks

> Use when CI checks fail unexpectedly, when preparing code for CI, or when encountering non-obvious build and pipeline behavior. Covers vendoring, hermetic builds, security scans, code generation checks, coverage, and webhook validation gotchas.

- Skill: `tomevault-io/ci-cd-quirks-4` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/ci-cd-quirks-4`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/ci-cd-quirks-4/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/ci-cd-quirks-4

---


# CI/CD Quirks and Gotchas

## Overview

Integration-service CI runs on Konflux (Tekton-based) for builds and GitHub Actions for linting/testing. Several non-obvious requirements catch developers off guard.

## When to Use

- CI failed and you don't understand why
- Preparing a change that touches dependencies, CRDs, or RBAC
- Understanding what the build pipeline actually does
- Debugging webhook validation behavior

## Must-Run Commands Before Push

| When you changed... | Run this | Why |
|---------------------|----------|-----|
| Any `go.mod` / `go.sum` | `go mod tidy && go mod vendor` | Vendor dir is committed; CI fails on drift |
| Types in `api/v1beta2/` | `make generate manifests` | DeepCopy and CRD/RBAC manifests must be in sync |
| Any `.go` file | `make fmt` | CI fails on uncommitted gofmt changes |
| `+kubebuilder:rbac` markers | `make manifests` | RBAC ClusterRole must reflect markers |

## CI Checks That Surprise People

### Vendoring (most common failure)
Go vendoring prunes non-Go files. After `go mod vendor`, CRD YAML files from dependencies are gone. `make download-crds` re-downloads them for tests. CI runs this automatically, but if you updated deps and didn't vendor, CI fails.

### RBAC Wildcard Check
CI greps `config/` for RBAC wildcards (`*`) and fails if found. Never use wildcard verbs or resource names in `+kubebuilder:rbac` markers.

### Code Generation Drift
CI runs `make generate manifests` and diffs the result. Any uncommitted generated files = failure. Always commit generated output.

### `go mod tidy` Drift
CI runs `go mod tidy` and checks for changes. If your `go.mod`/`go.sum` differ after tidy, CI fails.

## Build Pipeline Details

### PR Pipeline (`.tekton/integration-service-pull-request.yaml`)
- Injects `ENABLE_COVERAGE=true` — binary built with `-cover -covermode=atomic -tags=coverage`
- Hermetic build (network-isolated), deps prefetched via Cachi2
- Multi-platform: x86_64 + arm64
- Image tag: `on-pr-{{revision}}`
- Expires after 5 days, max 3 kept
- Security scans: Clair, Roxctl, Snyk SAST, ClamAV, ShellCheck, Unicode, RPM signature

### Push Pipeline (`.tekton/integration-service-push.yaml`)
- Regular build (no coverage) + extra `build-instrumented-image` task (with coverage)
- Image tag: `{{revision}}`
- Same security scans as PR

### Dockerfile Gotchas
- Builds **two binaries**: `manager` (controller) and `snapshotgc` (garbage collector)
- Both ship in the same image, entrypoint is `/manager`
- `ENABLE_COVERAGE` build arg switches between instrumented and production builds
- Base image: UBI9 minimal, runs as user 65532 (non-root)

## Webhook Validation Gotchas

| Rule | What happens if violated |
|------|------------------------|
| ITS name must be DNS-1035 | Webhook rejects with validation error (lowercase alphanumeric + hyphen, <63 chars) |
| `SNAPSHOT` param in ITS | Cannot be set manually — auto-injected by the service. Webhook rejects. |
| Git resolver: `url` vs `repo+org` | Must use one or the other, not both. Webhook rejects conflicting params. |
| Snapshot validator | Has `failurePolicy: Ignore` — validation failures pass through silently (by design) |

## Coverage

- Tool: codecov, flag: `unit-tests`
- Threshold: 3% drop from base commit allowed
- Ignores: `**/zz_generated*`, `e2e-tests/**`, vendor
- Patch coverage: informational only (non-blocking)

## Common Mistakes

| Problem | Root cause |
|---------|-----------|
| CI fails with "uncommitted changes" | Didn't run `make fmt`, `make generate manifests`, or `go mod tidy` |
| Build fails on missing deps | Didn't run `go mod vendor` after dependency change |
| RBAC check fails | Used wildcard `*` in kubebuilder RBAC marker |
| Coverage dropped | New code paths not covered by tests (check `cover.out`) |
| Webhook rejects valid-looking ITS | Name contains uppercase, underscore, or is >63 chars |

---
> Source: [konflux-ci/integration-service](https://github.com/konflux-ci/integration-service) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-06-15 -->