# Harbor Index Release

> Orchestrate a full Harbor Index release cut (build-push-pin, oracle gate, Hub publish, leaderboard cutover, image aliases, signed tag / GitHub Release) or a hotfix re-pin. Use when a maintainer asks to cut harbor-index vX.Y, ship a release, do a hotfix release, publish harbor-index, or run the release runbook. Does not cover post-release GPT evals or leaderboard submissions.

- Skill: `harbor-framework/harbor-index-release` (Agent Skill)
- Install (CLI): `npx skillmds@latest add harbor-framework/harbor-index-release`
- Raw SKILL.md: https://api.skillmd.com/api/skills/harbor-framework/harbor-index-release/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: harbor-framework (https://skillmd.com/u/harbor-framework)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/harbor-framework/harbor-index-release

---


# Harbor Index release

Drive the **real multi-step** release. There is **no** one-shot “trigger release” workflow.

| Automated | Manual / agent-orchestrated |
|---|---|
| `build-push-pin.yml` (`workflow_dispatch`) → builds, pins, pushes `release/X.Y` | Oracle gate, Hub publish, leaderboard cutover |
| `github-release.yml` (tag push `harbor-index-*`) → GitHub Release | Image aliases, GPG signed tag |

Canonical tooling detail (do not duplicate here):
- [scripts/README-build-push-pin.md](../../../scripts/README-build-push-pin.md)
- [`.github/workflows/build-push-pin.yml`](../../../.github/workflows/build-push-pin.yml)
- [`.github/workflows/github-release.yml`](../../../.github/workflows/github-release.yml)
- [leaderboard/SETUP.md](../../../leaderboard/SETUP.md)

**OUT OF SCOPE:** GPT evals, leaderboard submissions (`lb submit`), task-maintenance docs (`VERIFIER_TIMEOUTS.md`, `ADAPTATIONS.md`).

## Progress checklist

```
Release Progress:
- [ ] 0. Confirm version + mode with user (STOP)
- [ ] 1. Preconditions (main green / FROM pins)
- [ ] 2. build-push-pin (dispatch; execute=true needs STOP)
- [ ] 3. Oracle 5× + opt base-TN + oracle-gate attestation commit
- [ ] 4. Harbor publish (tasks → dataset) + commit dataset.toml (STOP)
- [ ] 5. Leaderboard create + cutover constants (STOP before merge)
- [ ] 6. Image aliases + signed tag + verify GitHub Release (STOP)
- [ ] 7. Report links / remaining follow-ups
```

## STOP / ask-user gates

Halt and get explicit approval before:

1. **Version / mode** — confirm `X.Y` (initial) vs `X.Y.Z` (hotfix) and target branch.
2. **`execute=true`** on `build-push-pin` (builds/pushes ~171 images; dry-run is default).
3. **Public Hub publish** (`harbor publish … --public`).
4. **GPG / signed tag** (`git tag -s`) and pushing `harbor-index-*`.
5. **Merging PRs** (leaderboard cutover to `main`, etc.).

Never invent a fake end-to-end workflow. Prefer dry-run / preview first.

## Auth needed

- `gh` with workflow dispatch + contents permissions
- Docker Hub login for local `alias` (CI uses `DOCKERHUB_*`)
- `harbor auth login` / `HARBOR_API_KEY` for publish + leaderboard create
- Modal + judge keys for oracle (`job-config.yaml` verifier envs)

## Step 0 — Confirm version

- **Initial:** version `X.Y`, branch `release/X.Y`, tag `harbor-index-X.Y` — must not already exist.
- **Hotfix:** version `X.Y.Z`, dispatch **from** existing `release/X.Y` after cherry-picks from `main`.

## Step 1 — Preconditions

```sh
git fetch origin --prune --tags
uv run scripts/build_push_pin.py lint --scope main   # initial only
gh run list --branch main --limit 10                 # CI green
```

- Every `FROM` must be digest-pinned on `main`; no `docker_image` pins on `main`.
- If FROMs are unpinned: `resolve-from-digests` → PR to `main` **before** cutting.
- Use a **clean worktree** of the release ref for oracle/publish/tag work.

## Step 2 — build-push-pin

Inputs: `version` (`X.Y` / `X.Y.Z`), `mode` (`initial`|`hotfix`), `execute` (`false` default).

Guards when `execute=true`:
- **initial:** from `main`; `release/X.Y` and tag `harbor-index-X.Y` must not exist.
- **hotfix:** from `release/<minor>`; version must be `X.Y.Z`.

**Initial** (from `main`):

```sh
gh workflow run build-push-pin.yml --ref main \
  -f version=X.Y -f mode=initial -f execute=true
```

**Hotfix** (from `release/X.Y`, after cherry-picks):

```sh
gh workflow run build-push-pin.yml --ref release/X.Y \
  -f version=X.Y.Z -f mode=hotfix -f execute=true
```

Monitor until success. Download **`digests-merged`** → keep as `digests.json` (needed for `oracle-gate` and `alias`).

Local bootstrap alternative: `uv run scripts/build_push_pin.py release --version X.Y --execute`

If a partial `initial` run already created `release/X.Y`, do **not** re-dispatch `mode=initial` — switch to `hotfix`.

## Step 3 — Oracle gate

From a checkout of `release/X.Y` (pinned images). Two halves are required:

1. Full **5×** oracle on **required** tasks (= all tasks minus `ORACLE_SKIP` in `scripts/build_push_pin.py`), Modal amd64, upload public:

```sh
# build include list from ORACLE_SKIP, then:
harbor run --job-name release/oracle-X.Y-… \
  -c job-config.yaml -p tasks \
  --include-task-name <each-required-task> … \
  -a oracle -e modal -k 5 -n 32 \
  --upload --public -y
```

2. **Base-TN** on all optimization / relative-speedup tasks (auto-discovered:
   `algotune-*`, `gso-speedup-*`, plus verifier markers). Each baseline must be
   **correct-but-slow**: pass equality / `is_solution`, fail **only** the
   relative speed bar (`reward==0`, `reward_pre_threshold < threshold`). A TN
   that fails correctness (e.g. GSO empty `nop` → `empty_patch`) is invalid.
   Modal amd64; storage ≤10240:

```sh
# AlgoTune: naive Task.solve overlay via -a oracle.
# GSO: comment-only no-op overlay via -a oracle (NOT empty nop).
# Prefer k=3 for AlgoTune; k=1 is fine for long GSO tasks.
uv run scripts/run_opt_base_tn.py run \
  --jobs-dir /data/eval-1.3/smoke/opt-base-tn-v2 \
  --env-file /data/harbor-adapters-experiments/.env \
  -e modal --upload
uv run scripts/run_opt_base_tn.py discover   # include list / methods
```

3. Attest and commit (STOP if either half fails):

```sh
uv run scripts/build_push_pin.py oracle-gate \
  --version X.Y --job <oracle-hub-uuid> \
  --base-tn-trials-in /data/eval-1.3/smoke/opt-base-tn-v2/trials.json \
  --base-tn-job <optional-hub-uuid> \
  --digests-in digests.json --execute
git add release-oracle/X.Y.json
git commit -m "release: attest vX.Y oracle gate"
git push origin release/X.Y
```

`--execute` requires `--job` (hub oracle UUID), `--base-tn-trials-in` from
`run_opt_base_tn.py` (schema `opt-base-tn-v2` with per-trial correctness), and
`--digests-in`. Optional `--base-tn-job` records a Hub UUID. Oracle
`--trials-in` stays dry-run/offline only. Attestation records all-`1.0` oracle
rewards **and** correct-but-slow base-TN evidence; `alias --execute` later
requires a hub-linked attestation whose `digest_set_sha256` matches the digest
map and whose `base_tn.requirement` is `correct-but-slow`.

## Step 4 — Harbor publish

**STOP** before `--public`. Order is mandatory: **tasks, then dataset**. No `--yes` on `harbor publish` — pipe confirmation. Pass `tasks/*` explicitly (nested dirs are not always auto-collected).

```sh
harbor init harbor-index/harbor-index --dataset --output-dir . \
  --description "Harbor Index: 80 agentic … (revision X.Y)." \
  --author "Harbor Framework"
harbor add tasks --scan --to dataset.toml
# confirm 80 unique digested tasks
printf 'y\n' | harbor publish tasks/* --tag X.Y --public --concurrency 20
printf 'y\n' | harbor publish . --tag X.Y --public --concurrency 20
```

Commit `dataset.toml` on the release branch **before** tagging so `github-release` can attach it. Resolve published dataset content hash from Hub for `DATASET_REF`.

## Step 5 — Leaderboard cutover

1. Create hub leaderboard `harbor-index-X-Y` (schema in [leaderboard/SETUP.md](../../../leaderboard/SETUP.md)):

```sh
harbor hub leaderboard create --config leaderboard.json --json
# or: curl POST …/leaderboard-create with HARBOR_API_KEY (SETUP.md)
```

2. Update on release branch **and** open a PR to `main` (**STOP** before merge):

| File | Field |
|---|---|
| `leaderboard/src/leaderboard/core/hub.py` | `DATASET_REF = "sha256:…"` |
| `leaderboard/src/leaderboard/ci/submit.py` | `LEADERBOARD_NAME = "harbor-index-X-Y"` |
| `leaderboard/SETUP.md` | Example `name` / `title` in `leaderboard.json` |
| Root `README.md` | Leaderboard URL query `leaderboard=harbor-index-X-Y` |
| `leaderboard/src/leaderboard/display_names.json` | Optional new display rows |

## Step 6 — Aliases + signed tag + GitHub Release

```sh
uv run scripts/build_push_pin.py alias \
  --version X.Y --digests-in digests.json --execute
# STOP — confirm GPG available; prefer signed tags:
git tag -s harbor-index-X.Y -m "harbor-index X.Y (frozen image set)"
git push origin harbor-index-X.Y
gh run list --workflow github-release.yml --limit 3
gh release view harbor-index-X.Y
```

`github-release.yml` uses `--verify-tag` (unsigned tags fail) and attaches `release-oracle/X.Y.json` + `dataset.toml` when present. Release notes need an explicit previous `harbor-index-*` tag (`previous_tag_name`) because divergent release branches are not linear ancestors (#64).

## Hotfix path

1. Land Dockerfile/source fix on `main`.
2. Cherry-pick onto `release/X.Y`.
3. Dispatch `build-push-pin` with `mode=hotfix`, version `X.Y.Z`, `--ref release/X.Y`.
4. Re-run oracle (at least touched required tasks; full gate before alias if attestation must rebind), re-publish if task digests changed, re-alias, tag `harbor-index-X.Y.Z` if shipping a tagged hotfix.

**Drift:** after cherry-pick, tree hashes intentionally diverge from old pins. Preflight may use `lint --scope release --allow-drift` when that flag exists; **aggregate** still requires drift-free `lint --scope release` after re-pin. If preflight fails on drift and `--allow-drift` is missing on the ref, stop and fix tooling before re-dispatching.

## Pitfalls

1. Divergent release branches → release-note baseline needs previous tag (#64).
2. `alias` without `--digests-in` → `--execute` refuses.
3. Publish order → tasks before dataset.
4. Evaluate frozen digests / `@X.Y`, not floating `main`.
5. Unsigned tags → `--verify-tag` fails.
6. Hotfix drift / Modal CPU variance can flake oracles (e.g. fastText `-march=native`).
7. Dirty worktrees → use a clean release worktree.
8. Missing GPG → stop and ask; do not silently fall back without approval.

## Done when

- `release/X.Y` has pins + `release-oracle/X.Y.json` + `dataset.toml`
- Hub has public `harbor-index/harbor-index@X.Y`
- Leaderboard `harbor-index-X-Y` exists; cutover PR ready/merged on `main`
- Alias tags stamped; GitHub Release published for `harbor-index-X.Y`