# Remediate Image Cves > Orchestrate end-to-end CVE remediation for the Linux CCM, CNM, and health-probe-proxy images on cloud-provider-azure master or a release-X.Y branch, including builds, repeated Trivy verification, checkpoint commits, cleanup, validation, push, and pull-request creation. Use when the user wants to verify or fix CVEs on master or a release branch, run the three-image CVE workflow, or open a CVE remediation PR for either supported target branch. - Skill: `kubernetes-sigs/remediate-image-cves` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add kubernetes-sigs/remediate-image-cves` - Raw SKILL.md: https://api.skillmd.com/api/skills/kubernetes-sigs/remediate-image-cves/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: kubernetes-sigs (https://skillmd.com/u/kubernetes-sigs) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/kubernetes-sigs/remediate-image-cves --- # Remediate Image CVEs Remediate actionable fixable vulnerabilities in the Linux CCM, CNM, and health-probe-proxy images for one input target branch. Require the caller to provide an input branch that is exactly `master` or matches `^release-[0-9]+\.[0-9]+$`. Operate only on the current checkout; callers may run this skill concurrently in separate worktrees for separate input branches. ## Sources of Truth Read these repository skills before taking any action: - `.agents/skills/build-images/SKILL.md` - `.agents/skills/fix-image-cves/SKILL.md` - `.agents/skills/sync-go-modules/SKILL.md` Follow those skills as the sources of truth for Makefile image builds and Trivy-based CVE remediation and for final repository-wide Go module consistency. Do not duplicate, override, or extend their commands. This skill supplies orchestration, lifecycle, Git, validation, cleanup, and pull-request policy only. If a dependency is missing or conflicts with this skill, stop and report the blocker. Before checking out the input branch, copy this complete skill directory and all three dependency skill directories into a unique run-scoped temporary directory outside the repository. Use that exact tooling snapshot for the entire run; do not depend on skill files present on the target branch. Invoke both image helpers and the module-sync helper with their explicit `--repo` input pointing at the current worktree. Remove the temporary tooling snapshot during final cleanup. ## Git Lock Discipline Run every orchestration-owned read-only Git inspection with command-scoped `GIT_OPTIONAL_LOCKS=0`. This includes status, diff, log, show, rev-parse, ls-files, ls-tree, and ref checks. The image-CVE and module-sync helpers enforce the same rule internally for their read-only Git subprocesses. Do not export this setting across the workflow or pass it to mutating Git commands. Fetch, checkout, branch or ref updates, staging, commits, and pushes must run serially with normal required locking. Before each mutation and after each long-running helper exits, inspect the current worktree Git directory for lock files. If an unexpected lock exists, stop and report its path, owner, size, modification time, and holder check when available. Never delete, rename, bypass, or work around it as part of this skill. ## Preconditions 1. Require a clean worktree before changing branches. Preserve all pre-existing work; never reset, discard, or overwrite it. 2. Fail fast if Git, GitHub CLI, Trivy, Docker with Buildx or Podman with native build support, or a local Go version compatible with the checked-out branch is unavailable. Before each CVE apply, also require an available local Go version compatible with the fresh plan's Go directive actions. Do not install or upgrade host software or enable automatic toolchain downloads. 3. Require both `git var GIT_AUTHOR_IDENT` and `git var GIT_COMMITTER_IDENT` to succeed before any build so checkpoint commits cannot fail after remediation has started. 4. Require `upstream` to contain the input branch and verify, before any build, that the current authenticated account can push to `origin` and open a pull request against `upstream`. Check repository permissions rather than hard-coding classic-token scope names. 5. Check for an existing open CVE-remediation pull request targeting the input branch. If one exists, stop and report its URL instead of creating a duplicate. 6. If this worktree already has unfinished `fix-image-cves` state, stop and report it instead of deleting evidence from an earlier run. ## Branch and Image Identity Validate that the input branch is exactly `master` or matches `^release-[0-9]+\.[0-9]+$`. Fetch and check out its current tip from `upstream`, using only a fast-forward update if a local branch already exists. Never hard-reset a local branch; stop if it cannot be fast-forwarded safely. Then create a branch named `cve-fix--`, for example `cve-fix-master-20260721T013000Z` or `cve-fix-release-1.36-20260721T013000Z`. Use the build skill with: - image registry: `local` - image tag: `cve---` - explicit make inputs: `ARCH=amd64` and `OUTPUT_TYPE=docker` - inherited make inputs removed: `OUTPUT_FLAG` and `BUILDX_EXTRA_FLAGS` Generate `` once as a lowercase UUID without hyphens and retain it for the complete run. The tag must be unique to the run so concurrent worktrees cannot overwrite, scan, or delete one another's verification images. Pass those make inputs through the build skill's `--set` and `--unset` options on every build, including rebuilds and final verification builds. These images are disposable and must never be pushed. Select one container runtime for the complete run. Respect an explicit `CONTAINER_CLI`; otherwise use the Makefile preference of Podman when available and Docker otherwise. Resolve and record its absolute path, then pass it through the build skill as `--set CONTAINER_CLI=` on every build. Do not build with one runtime and scan or clean up with another. After each build, use the selected runtime to resolve and record the canonical local image reference. Use that exact reference for Trivy scans, rescans, and cleanup. Podman may normalize `local/...` to `localhost/local/...`; do not assume the Makefile reference is also the canonical runtime reference. ## Verification Set Use this order and identity in every phase: | Image | Build alias | Makefile image reference | Module root | Dockerfile | |---|---|---|---|---| | CCM | `ccm` | `local/azure-cloud-controller-manager:` | `.` | `Dockerfile` | | CNM | `cnm` | `local/azure-cloud-node-manager:-linux-amd64` | `.` | `cloud-node-manager.Dockerfile` | | health-probe-proxy | `hpp` | `local/health-probe-proxy:` | `health-probe-proxy` | `health-probe-proxy/Dockerfile` | For every build, allow concurrent builds from other worktrees. Docker builds share a host-level Buildx builder; Podman uses its native builder. Pass the build helper's `--retry-transient-runtime-errors` option on every build. The build skill owns runtime-specific failure classification, health checks, bounded delay, exact retry execution, and the two-attempt limit. Do not run an additional outer retry. If the helper still returns a failure, preserve both attempts in the run summary and stop. ### Baseline Phase Establish the three-image baseline before mutating source: 1. Build all three images from the untouched input-branch source before running any `fix-image-cves apply` command. 2. For each baseline image, run `scan` and `plan` with its canonical image reference, module root, and Dockerfile. Record the complete plan, actionable findings, unsupported fixable findings, and residual risks outside the helper state before scanning the next image. 3. Treat Go toolchain findings and findings without a `FixedVersion` as residual risks. Treat any OTHER finding with a non-empty `FixedVersion` as an unsupported fixable finding and stop without applying changes or opening a PR, preserving that image's state. 4. The helper has one worktree-local state file. When continuing, run its `clean` command after recording each successful baseline result so the next image starts with empty state. If a baseline scan or plan fails, preserve that image's state and stop. If all three baseline plans contain zero Go-module actions, zero base-image actions, and zero unsupported fixable findings, exit successfully without committing, pushing, or opening a PR. Only residual findings may remain on this no-change path. ### Remediation Phase After the complete baseline is recorded, process images sequentially in the table order: 1. Rebuild the image from the current source and resolve its canonical runtime reference. Run a fresh `scan` and `plan`; do not apply the saved baseline plan because an earlier image may already have changed a shared module. 2. Review every fresh plan. If it contains Go directive actions, select a locally installed Go version at least as new as the highest target before apply, and keep `GOTOOLCHAIN=local`. Apply Go-module and directive fixes through `fix-image-cves`, including compatible pinned Go builder targets for every affected verification Dockerfile (both CCM and CNM for the root module). Follow the fix skill's `Dockerfile:builder` target policy; keep already-compatible builders unchanged. For a fixable runtime base-image finding, automatically replace only the digest of the existing registry, repository, and tag. If runtime base-image remediation would change the image family or tag, stop for review. Stop and report any other permission, judgment, or conflict-resolution requirement instead of guessing. 3. If the fresh plan contains no Go-module or base-image actions and no unsupported fixable findings, record its residual risks, run `clean`, and continue to the next image. Skip `apply`, file verification, remediation rebuild, and planned-key rescan for this zero-action plan. 4. If the fresh plan has actions, run `apply`, then file verification. Rebuild the image and run `verify --rescan` to prove the planned vulnerability keys disappeared. 5. Before starting a new scan, record the apply state's exact modified-file list and verification results; `scan` replaces prior apply and verify state. Then run a fresh `scan` and `plan` of the rebuilt image to detect newly introduced actionable or unsupported fixable findings. 6. After the fresh plan is reviewed, commit only the recorded modified files as the checkpoint for that cycle. Do not commit immediately after file verification. The fresh plan need not be empty for an intermediate cycle. Never reset or revert an earlier checkpoint, and keep all checkpoint commits in the final PR. 7. If the fresh plan has no remaining actions or unsupported fixable findings, record residual risks, run `clean`, and continue to the next image. If it has more actions, use that current plan for the next cycle. Allow at most three apply/verify/fresh-plan cycles per image. 8. If verification fails, an unsupported fixable finding appears, or actions remain after the third cycle, mark the run incomplete. Stop without pushing or opening a PR and preserve the current helper state, source changes, and local checkpoint commits for diagnosis. ## Final Gate After all remediation cycles: 1. From the clean checkpointed source, run `sync-go-modules` once to normalize every tracked module and the root vendor tree exactly as CI does. If the vendor dependency set changes, rerun it with `--allow-dirty` and `--update-vendor-licenses`, then run it once more with `--allow-dirty` to normalize the root module after license generation. Inspect the resulting diff and commit only expected `go.mod`, `go.sum`, `vendor/`, and `LICENSES/` paths as a final module-consistency checkpoint. Never absorb unrelated files. 2. Run `sync-go-modules --check-clean`. Do not continue unless it produces no diff. If vendor-license generation was needed, normalize modules again after that generation before running `--check-clean` because the upstream license tooling executes `go list -m all`. 3. Run the full repository unit-test suite with `make test-unit`. 4. Run `git diff --check` against the input branch. 5. Rebuild all three images from the final committed source. 6. Freshly scan and plan all three rebuilt images in table order. Record each result before scanning the next image; a new `scan` replaces the previous state, so no intermediate `clean` is required. Stop immediately on failure so the failing image's current state remains available. Verification is complete only when none has remaining Go-module or base-image actions and none has an unsupported fixable OTHER finding. Preserve findings without a fixed version and Go toolchain findings as residual risks. 7. Require a clean worktree containing only the checkpoint commits relative to the input branch. If any gate fails or actionable or unsupported fixable vulnerabilities remain, do not push or open a PR. Preserve the local branch, checkpoint commits, current CVE state, and a precise failure summary. ## Pull Request When source changes exist and every final gate passes: 1. Push the CVE-fix branch to `origin`. 2. Open a ready-for-review PR in `kubernetes-sigs/cloud-provider-azure` targeting the input branch. 3. Treat `.github/PULL_REQUEST_TEMPLATE.md` in the target checkout as authoritative. Start from its exact contents and preserve every section heading, the issue field, the fenced `release-note` and `docs` blocks, and any additional sections it contains. HTML guidance comments may be removed only after following their instructions. Never replace or drop repository template structure to match the skill asset; stop if the two cannot be merged without losing required content. 4. Use title `chore: fix cves for `. Read `assets/pull-request-template.md` from the run-scoped snapshot of this skill, and replace every placeholder. Under `What this PR does / why we need it`, use `Remediates actionable fixable vulnerabilities in the Linux CCM, CNM, and health-probe-proxy images for .`, followed by the populated results table and validation block from the asset. Under `Special notes for your reviewer`, use only the populated residual-risk details block from the asset. Do not use the asset as the complete PR body or copy its content into any other section. 5. Preserve the repository template's heading text and order. Before opening the PR, verify that every `####` heading from the repository template appears exactly once in the same order and that its `release-note` and `docs` fences remain. Use `/kind cleanup`, no issue closure unless the caller supplied one, and `NONE` in the release-note block. Replace the issue placeholder with `NONE` when no issue was supplied. Count vulnerability findings rather than grouped remediation actions in the baseline column. Use `None` instead of omitting an empty fix or residual-risk entry. For a shared root-module fix, identify the checkpoint that covers both CCM and CNM. ## Cleanup and Output In a final cleanup step on both success and failure, use the selected runtime to remove only the three recorded canonical references for this run's uniquely tagged local verification images. Ignore an image that was never created. Never run a system prune, remove shared base layers, clear Buildx cache, or delete images belonging to another run. Clean `fix-image-cves` state after complete success. Preserve it after failure. On the no-change path, clean the state and remove the unused local CVE-fix branch. Output: | Image | Baseline findings | Applied fixes | Final verification | Residual risks | |---|---|---|---|---| Then report the module-consistency result, unit-test result, checkpoint commits, cleanup result, and PR URL, or the precise reason no PR was created.