CI CD Pipeline

CI/CD Pipelines for AI-Coded Apps

LLM-generated code ships with predictable failure modes: plausible-but-wrong logic, silently-disabled tests, hallucinated APIs, secrets pasted inline, dependencies pulled from typosquats, and "works on my laptop" defaults. The pipeline is the only honest reviewer — if it doesn't block, it ships.

When to apply

• New repo getting its first CI workflow
• Existing pipeline that "works" but has obvious gaps (no SAST, no SBOM, no coverage gate)
• A bug or incident traced back to "should have been caught in CI"
• Migrating to a reusable / shared workflow across repos
• Standardising AI-coded projects on a baseline before scaling team

Six categories of gates

Every pipeline needs coverage of these. Items marked 🔴 are non-negotiable — they fail the build.

1. Security gates 🔴

2. Correctness & maintainability 🔴

3. Scalability & performance

4. Release & runtime safety

• Environment protection rules with manual approval on production.
• Zero-downtime migration policy: expand-migrate-contract enforced via lint rule (no DROP / ALTER TYPE / non-nullable-without-default in one release).
• Progressive delivery: canary or blue/green; automated rollback on SLO breach.
• Feature flags for AI-heavy paths — kill-switch beats redeploy.
• Post-deploy smoke tests + synthetic monitors as a required status in the deploy workflow.
• Runbook link required in release notes; PR template enforces it.

5. AI-specific gates

• Prompt / model registry versioned in git — treat prompts, system messages, model IDs like code: reviewed, diffed, tagged.
• Eval harness in CI — small deterministic eval suite runs on every PR that touches an AI path (agent, prompt, model, tool). Block regressions, same as tests.
• Cost + latency budgets per endpoint, reported in PR comment. LLM spend is now a CI concern.
• PII / prompt-injection lint on user-input paths; red-team fixtures in the eval set.
• Reproducibility: pin model versions (no gpt-4o-latest in prod); log model + prompt hash with every response.
• Data contracts for tool-calling: JSON Schema validation on every LLM function-call argument before it hits business logic.

6. Consistency & agent governance 🔴 (for AI-coded work)

LLM agents drift in conventions (layers, names, utils, logging) faster than they drift in security. These controls act on the input side — before CI even runs — so the agent writes code that matches the repo instead of inventing a new shape per feature.

Audit wiring: PR-blocking vs scheduled-issue-tracking

Audits run with two distinct wirings. A repo usually needs both. Pick the right one per audit type — the difference is which event triggers the workflow and what "failure" means.

A. PR-blocking gates (fail the build)

• Triggers: pull_request, push to protected branches.
• Job behaviour: scan only the PR diff (or the new state). If a new Critical/High finding lands, fail the build so the PR can't merge.
• Examples: gitleaks on staged content, semgrep on changed files, npm audit against the new lockfile, container scan on the PR-built image, schema-diff on the new OpenAPI doc.
• Why: stops new findings from landing. Targeted, actionable, owned by the PR author.

B. Scheduled audits (manage GitHub issues, never fail the build)

• Triggers: schedule: (e.g. weekly cron) + workflow_dispatch.
• Job behaviour:
  • Scan the whole codebase / lockfile / image set.
  • If findings exist → open or update a single labelled GitHub issue per audit category. Idempotent: same title + label → update; new run → bump the body with the latest report.
  • If findings are gone → close the previously-opened issue.
  • The job itself always exits 0. The issue is the signal.
• Why: scheduled-job failures create noise without an actionable PR. Issues are actionable and form a live dashboard. The /issues view with the audit label becomes the team's standing audit board.

Where to apply pattern B (not pattern A)

Use scheduled-issue-tracking for findings that:

• Already exist in the codebase (you don't want to gate the next unrelated PR on them).
• Need triage/decision rather than immediate fix (e.g. "outdated dep — upgrade across 3 services this sprint").
• Cover concerns that are about the repo as a whole, not the diff.

Specific audits to wire as scheduled issue-trackers:

• Dependency vulnerabilities (pip-audit, npm audit, osv-scanner) — across the full lockfile, not just diff.
• Outdated dependencies (npm outdated, pip list --outdated).
• License drift / compliance.
• Container base-image vulns (rebuild + scan weekly even if no app changes).
• Dead-code / unused-dep accumulation reports.
• SBOM diff against the previous release.
• Cyclomatic-complexity / duplicate-code budgets when ratcheting (report drift, don't break builds mid-sprint).

Issue lifecycle (idempotent)

issue title:   "Audit: <category> — <project>"
issue labels:  ["audit", "automated", "<category>"]

on each scheduled run:
  if findings:
    if existing open issue with this title+label: update body (replace)
    else: create new issue
  else:
    if existing open issue with this title+label: close with comment "✅ no findings on <date>"

Use a helper such as peter-evans/create-issue-from-file or a small gh issue wrapper. The board ends up with one row per audit category per project: open issues = current debt, closed issues = recently-resolved debt with provenance.

One-line summary

Canonical jobs

When implementing the gates, use a fixed set of job: names so workflows are interchangeable across repos and reusable workflow_call templates work without per-repo edits. Branch-protection "required status checks" reference these names — renaming a job silently breaks merge protection.

Always required (PR-gating, every repo)

Required when applicable

AI-specific (when repo has AI paths)

Release-only (on tag / release event)

Deploy (separate workflow, gated by required status checks)

Scheduled audits (separate workflow, exit 0 — see "Audit wiring" above)

Branch-protection required status checks

The minimum that must be green before merging to a protected branch:

pr-guards
secret-scan
lint
typecheck
test-unit
test-integration
coverage-gate
sast
sca
dead-code
build

Plus, when applicable to the repo:

migrate-check
schema-diff
docker-build-scan
ai-eval
smoke-postdeploy

Naming rules

• Lowercase, hyphenated. Matches GitHub's job-slug convention.
• One concern per job. Don't merge lint and typecheck into quality — branch protection becomes coarse and a typecheck failure reruns lint.
• needs: only what's actually consumed. Coverage genuinely needs tests; lint doesn't need typecheck. Excessive needs: serialises the pipeline for no benefit.
• Never rename a job without auditing branch protection. Required-status-check names are referenced by the GitHub API; a rename either silently skips the check or blocks every merge until the rule is updated.
• Reusable workflow ships these names verbatim. A repo consuming _ci.yml should not need to remap.

A reference reusable implementation lives in this template at .github/workflows/_ci.yml — downstream repos call it with uses: mechemsi/claude-template/.github/workflows/_ci.yml@<sha>. Language-neutral worked examples for these jobs are in ./examples/ — a parameterised _ci.yml plus standalone secret-scan, scheduled-audit, and PR-template patterns. Same canonical job names, swappable per ecosystem via workflow_call inputs.

Auditing an existing pipeline

Use this as a checklist when reviewing a project's CI. For each row, mark the project's status: ✅ in place, add to introduce, review if uncertain, — if not applicable.

Output the table marking each row's status, then prioritise the gaps using the adoption order below — don't try to fix all of them in one PR.

Minimum viable baseline — adoption order

Don't try to add all 60+ gates at once. This order maximises return per hour spent.

1. Gitleaks + push protection — one hour, stops the worst leaks.
2. Semgrep + language-specific SAST — one PR, pre-written rulesets.
3. Per-PR SCA (pip-audit, npm audit, osv-scanner) — upgrade weekly audit to per-PR.
4. Coverage gate + diff-cover + grep-in-diff for test-skip and TODO.
5. Trivy on Docker build + SBOM upload.
6. Reusable workflow_call pipeline shared across repos.
7. Preview environments + Lighthouse / k6 budgets.
8. AI eval harness + prompt/model registry (only if the repo has AI paths).
9. Drop AGENTS.md / CLAUDE.md + ADR seed + gold-standard file + PR template into every new project (or inherit from a template repo).
10. pre-commit + pinned runtimes — one PR per repo. Immediate CI-failure drop (~80%).
11. AI review bot as required check — pick one (Greptile for context, CodeRabbit for UX).
12. Nx / Turborepo in monorepos — affected graph + module-boundary lint.
13. Conventional Commits + semantic-release + Renovate — via reusable workflow_call.
14. Duplicate-code + complexity budgets — tune to current code, ratchet down.
15. Structured-logging lint + AI-disclosure PR template.
16. Agent sandboxing (harden-runner + ephemeral runner) for repos where agents run inside CI.

Everything above is boring, well-trodden, and low-cost. Skipping any of it on AI-coded apps is the same bet as shipping without tests — just slower to notice.

Common mistakes

• Treating SAST/SCA as warnings. Anything not gated is decoration. Security findings must fail the build (with a documented exception path).
• Coverage % without diff-cover. New code can land at 0% while overall % stays comfortable.
• Re-running flaky tests forever. Hides real bugs and trains the team to ignore red.
• @v4 action references — supply-chain risk. Pin SHAs.
• One mega ci.yml. Path-filter and split per surface so a docs-only PR doesn't run the backend test suite.
• Adopting all 60 gates day one — pipeline becomes a 30-minute slog, team starts disabling checks. Use the adoption order.
• No AGENTS.md / CLAUDE.md — agents drift the repo's conventions every PR; your reviewers spend their time on style, not logic.
• Self-hosted runners without harden-runner — runner becomes a foothold the moment a malicious dep lands.

Related

• security-review — companion skill for code-level security audit (this skill is pipeline-level).
• dependency-management rule — what gets installed; this skill is what gets checked.
• error-handling, logging-observability — paired with §6 lint rules (structured logs, trace IDs).
• 12-factor-app — release/runtime context for §4.
• testing-architecture — coverage gate and mutation testing depth come from here.
• db-migration-safety — pair with a CI gate that flags DROP/RENAME/NOT NULL/large-index DDL in migration diffs.
• feature-flags-and-rollout — CI matrix expansion for active rollouts; required-checks-green gate before flag flips.
• code-review-discipline — CI gates are the floor; review is the ceiling. Pipeline says "ready for review", not "ready to merge".
• performance-optimization — perf regression gates after a hot-path fix live in the pipeline.

Source: mechemsi/claude-template — distributed by TomeVault.