Prepare Release

Prepare a Release

This workflow prepares a new release of inspect_evals. It creates a release branch, collects changelog fragments, bumps the version tag, and opens a PR. After merge it tags the merge commit and creates a GitHub release.

Reference: PACKAGE_VERSIONING.md

Prerequisites

• The gh CLI must be authenticated
• You must have push access to the repository
• There must be changelog fragments in changelog.d/ (beyond .gitkeep and TEMPLATE.md)

Phase 1 — Prepare the release branch

1. Ensure main is up to date:

   git fetch origin
   git checkout main
   git merge --ff-only origin/main
   

   If the merge fails, stop and inform the user that their local main has diverged from origin/main.

2. Check for changelog fragments:

   List files in changelog.d/ excluding .gitkeep, TEMPLATE.md, and README.*. If there are no fragments, stop and tell the user there is nothing to release.

3. Determine the new version:

   1. Find the current version from the latest v* git tag:

      git tag --sort=-v:refname --list 'v*' | head -1
      

   2. Ask the user what kind of bump this is, presenting the semver table from PACKAGE_VERSIONING.md:

      Component	When to bump	Examples
      Major	Breaking changes	Removing an eval, API changes, scorer output format changes
      Minor	New features	Adding new evals, new task parameters, new utilities
      Patch	Bug fixes	Eval fixes, scorer fixes, dataset loading fixes

   3. Compute the new version string (e.g. 0.4.0, 0.3.107). Confirm with the user.

4. Create the release branch:

   The branch name is release-YYYY-MM-DD using today's date.

   git checkout -b release-YYYY-MM-DD
   

Phase 2 — Collect the changelog

1. Run scriv collect:

   uv run scriv collect --version <NEW_VERSION>
   

   This removes the individual fragment files from changelog.d/ and prepends a new section to CHANGELOG.md.

2. Present the changelog diff to the user for review:

   git diff CHANGELOG.md
   

   Tell the user: "Please review the collected changelog above. You can edit CHANGELOG.md directly — let me know when you are satisfied, or tell me what changes to make."

   Do not proceed until the user confirms the changelog is ready.

3. Normalise the collected changelog:

   changelog.d/ fragments are excluded from the mdformat and markdownlint-fix pre-commit hooks (fragments are partial documents), so a hand-edit during review — or an unusual fragment — can leave CHANGELOG.md in a state those hooks would rewrite. That turns the Markdown Lint and README check jobs red on main after the release PR merges (see #2225, #2329). Run the hooks over CHANGELOG.md, then confirm a second pass is clean:

   uv run --group dev pre-commit run mdformat --files CHANGELOG.md || true
   uv run --group dev pre-commit run markdownlint-fix --files CHANGELOG.md || true
   # A second pass must be clean: applying fixes is fine, failing to
   # converge means a real error that would still break main.
   uv run --group dev pre-commit run mdformat --files CHANGELOG.md
   uv run --group dev pre-commit run markdownlint-fix --files CHANGELOG.md
   

   If the second pass still reports changes, resolve them before committing.

4. Stage and commit:

   git add CHANGELOG.md changelog.d/
   git commit -m "Prepare release v<NEW_VERSION>"
   

Phase 3 — Push and open a PR

1. Push the branch:

   git push -u origin release-YYYY-MM-DD
   

2. Open a draft PR:

   Extract the new version's section from CHANGELOG.md (everything from the version heading up to but not including the next ## heading). Use this as the PR body:

   gh pr create --draft \
     --title "Release v<NEW_VERSION>" \
     --body "<EXTRACTED_CHANGELOG_SECTION>"
   

   Important: The PR title must start with Release v (e.g. Release v0.4.0) — this is how the release-on-merge.yml workflow identifies release PRs.

   Tell the user the PR URL and that it is in draft. They should mark it ready for review when appropriate.

Phase 4 — After the PR is merged (automated)

Once the release PR is merged into main, the .github/workflows/release-on-merge.yml GitHub Actions workflow automatically:

1. Tags the merge commit with v<NEW_VERSION>
2. Creates a GitHub release with the changelog section as the body
3. Builds and publishes the package to PyPI
4. Notifies Slack

No manual action is required. The package version is derived from the new tag by setuptools_scm.

Notes

• The package version is derived entirely from git tags via setuptools_scm — there is no version file to edit.
• If something goes wrong mid-workflow, the release branch can be deleted and the process restarted.
• The v prefix on tags is required (e.g. v0.3.107, not 0.3.107).
• The weekly-release.yml workflow can also automate Phases 1–3 on a schedule or via manual dispatch, creating the release branch and draft PR for you.