Gh Release Ship Loop

gh-release-ship-loop

The canonical 10-phase loop for shipping a release on a GitHub-backed project with CI deploying to one or more hosting platforms (Vercel for web, Fly for backend, Render, Cloudflare Workers, etc.).

This is the META-SKILL that bundles version-bump + CHANGELOG + lint + test + commit + push + CI-watch + live-verify + failure-memory. Use INSTEAD of trying to remember the order; the order is the skill.

When to invoke

• Any time you're about to git commit && git push something that will trigger CI
• Hotfixes (fix-only, no feature work)
• Feature ships (one or more commits → one PR)
• Multi-hop ship trains (the failure-mode case this skill was built for)
• Recovery operations: cherry-pick onto fresh branch, rebase + drop-redundant-commits, force-with-lease

When NOT to invoke

• Local-only experiments that won't touch CI
• Docs-only PRs (probably skip the test sweep)
• Dependabot auto-PRs (separate skill flow)

The 10-phase loop

1. CODE CHANGE           → make the actual fix / feature
2. VERSION BUMP          → bump in every site (typically 3-4)
3. CHANGELOG             → entry in CHANGELOG.md + public /changelog page (if applicable)
4. LINT + FORMAT         → manually run black / ruff / eslint / tsc on changed files
5. TEST SWEEP            → exact pytest / vitest invocation the CI test job uses
6. COMMIT                → conventional-commits, lowercase subject, body ≤100 chars
7. PUSH                  → main directly OR feature branch + PR
8. CI WATCH              → gh run list/view to completion
9. LIVE VERIFY           → curl the deployed surface; confirm version live
10. FAILURE MEMORY       → if anything broke, encode it as a feedback memory

Each phase has gotchas that cost real session time when skipped.

Phase 1 — CODE CHANGE

Standard: edit the file, run the change.

Gotcha: Edit tool returns "successfully updated" even when the diff didn't apply (CRLF/whitespace mismatch on small JSON/config files). Always grep-verify after Edit on:

• package.json, tsconfig.json, settings JSON
• generated files
• files with non-LF line endings

Pattern: after every Edit, run grep -n "<new content>" <file> to confirm the diff actually landed rather than silently no-op'ing.

Phase 2 — VERSION BUMP

Find every site where the version string lives:

# Python project
grep -rnE '"X\.Y\.Z"' src/ app/ api/

# JS project
cat package.json | jq .version
grep -rnE '"version": ".*"' .

# Multi-language monorepo
git grep -lE '_version|VERSION_STRING|"version"'

Common sites:

• package.json (web)
• api/app/main.py (FastAPI) — typically 3 sites: version=, _app_version =, _ROOT_RESPONSE_BODY["version"]
• api/app/mcp_gateway.py if there's an MCP gateway constant
• Generated SDK clients
• Docker labels

Gotcha: missing one site → tests that assert version-consistency fail at CI. The release-ship skill for that specific project should list every site explicitly.

Bash one-liner pattern (avoid sed bugs):

# Use sed only for simple drop-in version replaces; verify with grep after
sed -i 's/"X.Y.Z"/"A.B.C"/g' file.py
grep -nE '"X\.Y\.Z"|"A\.B\.C"' file.py  # all should be A.B.C, none X.Y.Z

Phase 3 — CHANGELOG

CHANGELOG.md entry following Keep a Changelog format:

## [X.Y.Z] — YYYY-MM-DD

### Added | Changed | Fixed | Deprecated | Removed | Security

- **Hotfix:** description with context. Include Sentry issue ID if applicable.

Gotcha: if your project has a CUSTOMER-FACING /changelog page (e.g. Next.js MDX or a TypeScript array of ChangelogEntry objects), update that ALSO. A CI test will fail if the public page lags CHANGELOG.md.

For a locale-aware Next.js project: <web-app>/src/app/[locale]/(unauth)/changelog/page.tsx + a matching changelog-entry id. For other projects, look for a public-changelog-sync check in CI.

Phase 4 — LINT + FORMAT (manually)

Hard rule: the pre-commit lint-staged hook DOES NOT catch everything. Two paths bypass it:

1. cat >> file.py <<EOF heredoc — writes through filesystem before git knows; lint-staged only inspects staged files at commit time
2. Edit tool from an LLM agent — same path; bypasses git stage

Both produce CI lint failures that cost a full CI cycle (~15 min) to discover + fix.

Manual recovery — run BEFORE push:

# Python (typical pattern)
black --check --line-length 100 <changed-files>
ruff check <changed-files>

# TypeScript / Next.js
cd <web-app> && npm run check-types
cd <web-app> && npm run lint -- <changed-files>

# Other
prettier --check <changed-files>

If black --check says "would reformat", run without --check to apply, then re-stage and re-commit.

Lesson: a heredoc-written file can bypass lint-staged's staged-file diffing if it isn't re-staged after formatting — always re-stage and re-commit after an auto-format pass.

Phase 5 — TEST SWEEP (W14 cascade prevention)

The CI test job runs with a specific env config that local pytest doesn't match by default. If you push without running the EXACT CI invocation locally, you risk a test-debt cascade where:

• Local pytest passes (different env)
• CI fails on push
• Repeat 2-3x before discovery

The fix: find the CI test job's exact invocation in .github/workflows/<deploy>.yml and replicate it:

# Example (Python/FastAPI)
PYTHONPATH=. \
  MCP_GATEWAY_ENABLED=0 \
  UPSTASH_REDIS_REST_URL=mock \
  DATABASE_URL=sqlite+aiosqlite:///test.db \
  pytest api/tests -q --ignore=api/tests/e2e --ignore=api/tests/integration

# Example (Next.js + Vitest)
cd <web-app> && npm test

# Example (Cargo)
cargo test --workspace

Lesson (W14): always run the full CI-equivalent test sweep locally before pushing, not just the changed-file subset — a partial local run misses cascade failures CI will catch.

Phase 6 — COMMIT

Conventional-commits — every commit message must follow the project's commitlint config:

fix(scope): subject in lowercase

Body paragraph here. Lines under 100 characters wide.
Body lines can wrap — no double-space at end of paragraph
needed.

Co-Authored-By: Claude Opus 4.X <noreply@anthropic.com>

Hard rules:

• Subject MUST be lowercase (commitlint default subject-case)
• Subject MUST start with a type (feat, fix, chore, docs, style, refactor, test, build, ci, perf, revert)
• Body lines ≤100 chars (most commitlint configs enforce body-max-line-length)
• NO --no-verify (skips signing AND lint hooks; never authorized unless explicit operator directive)
• NO --amend on a PUSHED commit (rewriting public history)
• Prefer new commits over amending when feasible

Pattern for multi-line commit message via heredoc:

git commit -m "$(cat <<'EOF'
fix(scope): one-line subject in lowercase

Detailed body here. Multiple paragraphs are fine.
Lines wrap at ~100 chars.

Co-Authored-By: Claude Opus 4.X <noreply@anthropic.com>
EOF
)"

Phase 7 — PUSH

To main directly — only authorized for:

• Hotfixes (the bug is in prod RIGHT NOW)
• Documentation-only changes
• Solo-developer projects with no PR review requirement

Via PR + merge — the default for everything else:

git switch -c feat/<descriptive-name>
git push -u origin feat/<descriptive-name>
gh pr create --title "feat: <subject>" --body "..."
# After CI green:
gh pr merge <N> --squash --delete-branch

Force-push policy:

• --force-with-lease is SAFER than --force (refuses if remote has unseen commits)
• ONLY use it on YOUR own feature branch
• NEVER on main/master/develop (warn the user if they request it)

Phase 8 — CI WATCH

Always watch CI through to completion. Never stop at "CI is firing."

Pattern for foreground watch:

# Get latest run id
gh run list --workflow=deploy.yml --branch=main --limit 1 --json databaseId,status,conclusion

# Watch jobs progress
gh run view <id> --json jobs --jq '.jobs[] | {name, status, conclusion}'

# Tail logs from a failing job
gh run view <id> --log-failed | tail -80

Pattern for background watch:

# Monitor with line-by-line emit until all checks settle
prev=""
while true; do
  s=$(gh pr checks 123 --json name,bucket)
  cur=$(jq -r '.[] | select(.bucket!="pending") | "\(.name): \(.bucket)"' <<<"$s" | sort)
  comm -13 <(echo "$prev") <(echo "$cur")
  prev=$cur
  jq -e 'all(.bucket!="pending")' <<<"$s" >/dev/null && break
  sleep 30
done

Lesson: monitor CI checks to actual completion (all buckets settled), not just the first "passing" signal — a check can flip from pending to failing after you've already stopped watching.

Phase 9 — LIVE VERIFY

After CI deploys, ALWAYS confirm the new version is live on the production surface.

# For an API serving a version JSON
curl -s https://api.example.com/ | jq -r .version

# For a Next.js app
curl -sI https://example.com/ | head -1
curl -s https://example.com/ | grep -oE 'data-version="[^"]+"'

# For a CLI
gh release view --json tagName

# For a Docker image
docker pull example/app:latest && docker inspect example/app:latest --format='{{.Config.Labels.version}}'

If the version doesn't match what you just shipped → investigate before declaring done. The deploy may be in flight, OR the deploy may have failed silently.

Phase 10 — FAILURE MEMORY (if anything broke)

If ANY phase 1-9 failed in a way that took >5 min to diagnose, encode a feedback memory before moving on.

Memory file location (project-scoped):

~/.claude/projects/<project-slug>/memory/feedback_<short-name>.md

Memory file shape:

---
name: <one-line rule>
description: <when this rule applies + receipt of the failure>
type: feedback
---

**Rule:** what to do (or not do) next time.

**Why:** the failure mode + how it cost time.

**How to apply:**

1. specific actionable step
2. specific actionable step

**Receipt:** the exact failure from this session with timestamps + commit SHAs.

**Companion memories:** related encoded lessons.

**See also:** relevant skills, docs, runbook links.

Then add a one-line index entry to MEMORY.md:

- [short-name-kebab-case](feedback_<short-name>.md) — one-sentence summary

Lesson (from a multi-hop production ship train — several sequential hotfixes on the same feature): a feature that spans multiple layers (API, frontend, worker, config) needs an end-to-end smoke test across ALL layers before declaring it done — passing unit tests on each layer individually is not sufficient proof.

Recovery patterns

When the basic 10-phase loop falls apart mid-flight, these are the recovery moves.

Cherry-pick onto a fresh branch from main

When a commit lives on the wrong branch (e.g. a hotfix mistakenly committed to a feature branch):

# Find the orphaned commit
git reflog | grep "<message keyword>"
# Cherry-pick onto fresh branch from main
git switch -c hotfix/<name> origin/main
git cherry-pick <sha>
# Fix any conflicts; add version bumps + CHANGELOG
git push origin hotfix/<name>:main  # if you have direct push perms

Rebase + drop redundant commits

When a feature branch has commits that already landed on main via cherry-pick or earlier merge:

git switch feat/<branch>
git rebase origin/main
# Git will say "Could not apply <sha>" — its changes are already in main
git rebase --skip
# Continue until rebase completes
git push --force-with-lease origin feat/<branch>

The PR auto-updates after the force-push.

Branch backup before destructive operations

ALWAYS push a backup branch before git reset --hard or git rebase -i:

git push origin <current-sha>:refs/heads/backup/<descriptive-name>
# Now do the destructive thing; if it goes wrong, you can fetch the backup

Encoded after a near-loss of subagent work during a hotfix recovery (the subagent's commit was orphaned by my branch switch; recovered via reflog + push to backup branch).

CI-failure debugging — Sentry first

When a deployed endpoint 500s, do NOT start with flyctl logs / heroku logs / cloud logs. Logs are tail-capped (~100 lines on Fly) and Starlette / Express / Rails middleware-chain tracebacks routinely exceed that window with the actual exception at the BOTTOM.

Sentry-first:

# Via Sentry MCP
mcp__claude_ai_Sentry__search_issues organizationSlug="<org>" projectSlugOrId="<project>" query="is:unresolved firstSeen:-15m <keyword>"

The Sentry frame is source-mapped + has the exact file:line of the actual exception. Saves 10-15 min vs grepping flyctl tail output.

Lesson: tail-capped log commands (flyctl logs, heroku logs, cloud logs) routinely truncate before the actual exception — check Sentry (or your error tracker) first, not raw log tails.

Cross-cutting hard rules

1. One commit per finding. Don't bundle a doc fix with a lint cleanup with a real code change. Three commits, three diffs, three reverts available.
2. Never push to main/master without permission unless explicitly authorized for hotfixes / docs-only / solo project.
3. Never --no-verify (skips signing AND lint hooks) without explicit user request.
4. Never git reset --hard / git push --force without a backup branch first. Save the SHA.
5. Never delete files / dirs that look unfamiliar. Investigate first.
6. Never disable hooks (HUSKY=0, core.hooksPath=/dev/null) as a shortcut.
7. Conventional-commits subject is LOWERCASE — commitlint default subject-case rejects feat: Fix Foo (start-case) and feat: FIX FOO (upper-case).

Tools used

Bash with the gh + git + curl + jq + sed + grep CLIs. No external dependencies beyond standard developer setup.

Anti-patterns to skip

• "Just push and see if CI passes" — costs a full CI cycle (~15 min). Run lint + tests locally first.
• "I'll fix the CHANGELOG later" — public-changelog-sync tests fail; later = next-commit churn.
• "The merge looks clean, ship it" — verify version is actually live via curl, not just CI green.
• "I'll skip the test sweep, it's a tiny change" — the next dev who runs the full sweep inherits your debt.
• "Force-push to main to undo a mistake" — never. Revert commit on top.

Project-specific variants

This is the GENERIC loop. Specific projects layer their own gates on top:

• A project-local release-ship skill: encodes the project's specific version sites + a test-sweep cascade gate + an endpoint-inventory test
• A research/paper-writing project: a project-local paper-drafting skill (different gates — peer-review rubric, noise floor)
• Other projects: create a project-local release-ship.md skill that subclasses this one, listing the specific version sites + lint commands + CI invocation.

See also

• Phase 4 lesson: heredoc-written files can bypass lint-staged — re-stage after formatting.
• Phase 5 lesson: always run the full CI-equivalent test sweep before pushing.
• Phase 8 lesson: monitor CI checks to actual completion, not the first green signal.
• Phase 10 lesson: check the error tracker (e.g. Sentry) before tail-capped logs.
• Meta-lesson: why this loop matters — a multi-hop ship train on one feature is a sign the e2e smoke test was skipped.
• Adjacent lesson: TypeScript types can lie about the actual wire shape of an API response — verify against the real payload, not just the type.
• Adjacent lesson: an endpoint-inventory test can silently block new routes from being registered — check it explicitly when adding a route (FastAPI-specific).

This skill was born from a multi-hop production ship train on a single feature:

Several hotfixes shipped sequentially on the same feature. Each fix took 1 of the 10 phases above to surface. The loop wasn't documented as a skill until phase 10 surfaced enough times that the META-pattern became obvious. Now it's a skill.