CI/CD Quirks and Gotchas
Overview
Integration-service CI runs on Konflux (Tekton-based) for builds and GitHub Actions for linting/testing. Several non-obvious requirements catch developers off guard.
When to Use
- CI failed and you don't understand why
- Preparing a change that touches dependencies, CRDs, or RBAC
- Understanding what the build pipeline actually does
- Debugging webhook validation behavior
Must-Run Commands Before Push
| When you changed... |
Run this |
Why |
Any go.mod / go.sum |
go mod tidy && go mod vendor |
Vendor dir is committed; CI fails on drift |
Types in api/v1beta2/ |
make generate manifests |
DeepCopy and CRD/RBAC manifests must be in sync |
Any .go file |
make fmt |
CI fails on uncommitted gofmt changes |
+kubebuilder:rbac markers |
make manifests |
RBAC ClusterRole must reflect markers |
CI Checks That Surprise People
Vendoring (most common failure)
Go vendoring prunes non-Go files. After go mod vendor, CRD YAML files from dependencies are gone. make download-crds re-downloads them for tests. CI runs this automatically, but if you updated deps and didn't vendor, CI fails.
RBAC Wildcard Check
CI greps config/ for RBAC wildcards (*) and fails if found. Never use wildcard verbs or resource names in +kubebuilder:rbac markers.
Code Generation Drift
CI runs make generate manifests and diffs the result. Any uncommitted generated files = failure. Always commit generated output.
go mod tidy Drift
CI runs go mod tidy and checks for changes. If your go.mod/go.sum differ after tidy, CI fails.
Build Pipeline Details
PR Pipeline (.tekton/integration-service-pull-request.yaml)
- Injects
ENABLE_COVERAGE=true — binary built with -cover -covermode=atomic -tags=coverage
- Hermetic build (network-isolated), deps prefetched via Cachi2
- Multi-platform: x86_64 + arm64
- Image tag:
on-pr-{{revision}}
- Expires after 5 days, max 3 kept
- Security scans: Clair, Roxctl, Snyk SAST, ClamAV, ShellCheck, Unicode, RPM signature
Push Pipeline (.tekton/integration-service-push.yaml)
- Regular build (no coverage) + extra
build-instrumented-image task (with coverage)
- Image tag:
{{revision}}
- Same security scans as PR
Dockerfile Gotchas
- Builds two binaries:
manager (controller) and snapshotgc (garbage collector)
- Both ship in the same image, entrypoint is
/manager
ENABLE_COVERAGE build arg switches between instrumented and production builds- Base image: UBI9 minimal, runs as user 65532 (non-root)
Webhook Validation Gotchas
| Rule |
What happens if violated |
| ITS name must be DNS-1035 |
Webhook rejects with validation error (lowercase alphanumeric + hyphen, <63 chars) |
SNAPSHOT param in ITS |
Cannot be set manually — auto-injected by the service. Webhook rejects. |
Git resolver: url vs repo+org |
Must use one or the other, not both. Webhook rejects conflicting params. |
| Snapshot validator |
Has failurePolicy: Ignore — validation failures pass through silently (by design) |
Coverage
- Tool: codecov, flag:
unit-tests
- Threshold: 3% drop from base commit allowed
- Ignores:
**/zz_generated*, e2e-tests/**, vendor
- Patch coverage: informational only (non-blocking)
Common Mistakes
| Problem |
Root cause |
| CI fails with "uncommitted changes" |
Didn't run make fmt, make generate manifests, or go mod tidy |
| Build fails on missing deps |
Didn't run go mod vendor after dependency change |
| RBAC check fails |
Used wildcard * in kubebuilder RBAC marker |
| Coverage dropped |
New code paths not covered by tests (check cover.out) |
| Webhook rejects valid-looking ITS |
Name contains uppercase, underscore, or is >63 chars |
Source: konflux-ci/integration-service — distributed by TomeVault.
1---2name: ci-cd-quirks-43description: Use when CI checks fail unexpectedly, when preparing code for CI, or when encountering non-obvious build and pipeline behavior. Covers vendoring, hermetic builds, security scans, code generation checks, coverage, and webhook validation gotchas.4---56# CI/CD Quirks and Gotchas78## Overview910Integration-service CI runs on Konflux (Tekton-based) for builds and GitHub Actions for linting/testing. Several non-obvious requirements catch developers off guard.1112## When to Use1314- CI failed and you don't understand why15- Preparing a change that touches dependencies, CRDs, or RBAC16- Understanding what the build pipeline actually does17- Debugging webhook validation behavior1819## Must-Run Commands Before Push2021| When you changed... | Run this | Why |22|---------------------|----------|-----|23| Any `go.mod` / `go.sum` | `go mod tidy && go mod vendor` | Vendor dir is committed; CI fails on drift |24| Types in `api/v1beta2/` | `make generate manifests` | DeepCopy and CRD/RBAC manifests must be in sync |25| Any `.go` file | `make fmt` | CI fails on uncommitted gofmt changes |26| `+kubebuilder:rbac` markers | `make manifests` | RBAC ClusterRole must reflect markers |2728## CI Checks That Surprise People2930### Vendoring (most common failure)31Go vendoring prunes non-Go files. After `go mod vendor`, CRD YAML files from dependencies are gone. `make download-crds` re-downloads them for tests. CI runs this automatically, but if you updated deps and didn't vendor, CI fails.3233### RBAC Wildcard Check34CI greps `config/` for RBAC wildcards (`*`) and fails if found. Never use wildcard verbs or resource names in `+kubebuilder:rbac` markers.3536### Code Generation Drift37CI runs `make generate manifests` and diffs the result. Any uncommitted generated files = failure. Always commit generated output.3839### `go mod tidy` Drift40CI runs `go mod tidy` and checks for changes. If your `go.mod`/`go.sum` differ after tidy, CI fails.4142## Build Pipeline Details4344### PR Pipeline (`.tekton/integration-service-pull-request.yaml`)45- Injects `ENABLE_COVERAGE=true` — binary built with `-cover -covermode=atomic -tags=coverage`46- Hermetic build (network-isolated), deps prefetched via Cachi247- Multi-platform: x86_64 + arm6448- Image tag: `on-pr-{{revision}}`49- Expires after 5 days, max 3 kept50- Security scans: Clair, Roxctl, Snyk SAST, ClamAV, ShellCheck, Unicode, RPM signature5152### Push Pipeline (`.tekton/integration-service-push.yaml`)53- Regular build (no coverage) + extra `build-instrumented-image` task (with coverage)54- Image tag: `{{revision}}`55- Same security scans as PR5657### Dockerfile Gotchas58- Builds **two binaries**: `manager` (controller) and `snapshotgc` (garbage collector)59- Both ship in the same image, entrypoint is `/manager`60- `ENABLE_COVERAGE` build arg switches between instrumented and production builds61- Base image: UBI9 minimal, runs as user 65532 (non-root)6263## Webhook Validation Gotchas6465| Rule | What happens if violated |66|------|------------------------|67| ITS name must be DNS-1035 | Webhook rejects with validation error (lowercase alphanumeric + hyphen, <63 chars) |68| `SNAPSHOT` param in ITS | Cannot be set manually — auto-injected by the service. Webhook rejects. |69| Git resolver: `url` vs `repo+org` | Must use one or the other, not both. Webhook rejects conflicting params. |70| Snapshot validator | Has `failurePolicy: Ignore` — validation failures pass through silently (by design) |7172## Coverage7374- Tool: codecov, flag: `unit-tests`75- Threshold: 3% drop from base commit allowed76- Ignores: `**/zz_generated*`, `e2e-tests/**`, vendor77- Patch coverage: informational only (non-blocking)7879## Common Mistakes8081| Problem | Root cause |82|---------|-----------|83| CI fails with "uncommitted changes" | Didn't run `make fmt`, `make generate manifests`, or `go mod tidy` |84| Build fails on missing deps | Didn't run `go mod vendor` after dependency change |85| RBAC check fails | Used wildcard `*` in kubebuilder RBAC marker |86| Coverage dropped | New code paths not covered by tests (check `cover.out`) |87| Webhook rejects valid-looking ITS | Name contains uppercase, underscore, or is >63 chars |8889---90> Source: [konflux-ci/integration-service](https://github.com/konflux-ci/integration-service) — distributed by [TomeVault](https://tomevault.io).91<!-- tomevault:4.0:skill_md:2026-06-15 -->
tomevault-io