Harbor Index Release

Harbor Index release

Drive the real multi-step release. There is no one-shot “trigger release” workflow.

Canonical tooling detail (do not duplicate here):

• scripts/README-build-push-pin.md
• .github/workflows/build-push-pin.yml
• .github/workflows/github-release.yml
• leaderboard/SETUP.md

OUT OF SCOPE: GPT evals, leaderboard submissions (lb submit), task-maintenance docs (VERIFIER_TIMEOUTS.md, ADAPTATIONS.md).

Progress checklist

Release Progress:
- [ ] 0. Confirm version + mode with user (STOP)
- [ ] 1. Preconditions (main green / FROM pins)
- [ ] 2. build-push-pin (dispatch; execute=true needs STOP)
- [ ] 3. Oracle 5× + opt base-TN + oracle-gate attestation commit
- [ ] 4. Harbor publish (tasks → dataset) + commit dataset.toml (STOP)
- [ ] 5. Leaderboard create + cutover constants (STOP before merge)
- [ ] 6. Image aliases + signed tag + verify GitHub Release (STOP)
- [ ] 7. Report links / remaining follow-ups

STOP / ask-user gates

Halt and get explicit approval before:

1. Version / mode — confirm X.Y (initial) vs X.Y.Z (hotfix) and target branch.
2. execute=true on build-push-pin (builds/pushes ~171 images; dry-run is default).
3. Public Hub publish (harbor publish … --public).
4. GPG / signed tag (git tag -s) and pushing harbor-index-*.
5. Merging PRs (leaderboard cutover to main, etc.).

Never invent a fake end-to-end workflow. Prefer dry-run / preview first.

Auth needed

• gh with workflow dispatch + contents permissions
• Docker Hub login for local alias (CI uses DOCKERHUB_*)
• harbor auth login / HARBOR_API_KEY for publish + leaderboard create
• Modal + judge keys for oracle (job-config.yaml verifier envs)

Step 0 — Confirm version

• Initial: version X.Y, branch release/X.Y, tag harbor-index-X.Y — must not already exist.
• Hotfix: version X.Y.Z, dispatch from existing release/X.Y after cherry-picks from main.

Step 1 — Preconditions

git fetch origin --prune --tags
uv run scripts/build_push_pin.py lint --scope main   # initial only
gh run list --branch main --limit 10                 # CI green

• Every FROM must be digest-pinned on main; no docker_image pins on main.
• If FROMs are unpinned: resolve-from-digests → PR to main before cutting.
• Use a clean worktree of the release ref for oracle/publish/tag work.

Step 2 — build-push-pin

Inputs: version (X.Y / X.Y.Z), mode (initial|hotfix), execute (false default).

Guards when execute=true:

• initial: from main; release/X.Y and tag harbor-index-X.Y must not exist.
• hotfix: from release/<minor>; version must be X.Y.Z.

Initial (from main):

gh workflow run build-push-pin.yml --ref main \
  -f version=X.Y -f mode=initial -f execute=true

Hotfix (from release/X.Y, after cherry-picks):

gh workflow run build-push-pin.yml --ref release/X.Y \
  -f version=X.Y.Z -f mode=hotfix -f execute=true

Monitor until success. Download digests-merged → keep as digests.json (needed for oracle-gate and alias).

Local bootstrap alternative: uv run scripts/build_push_pin.py release --version X.Y --execute

If a partial initial run already created release/X.Y, do not re-dispatch mode=initial — switch to hotfix.

Step 3 — Oracle gate

From a checkout of release/X.Y (pinned images). Two halves are required:

1. Full 5× oracle on required tasks (= all tasks minus ORACLE_SKIP in scripts/build_push_pin.py), Modal amd64, upload public:

# build include list from ORACLE_SKIP, then:
harbor run --job-name release/oracle-X.Y-… \
  -c job-config.yaml -p tasks \
  --include-task-name <each-required-task> … \
  -a oracle -e modal -k 5 -n 32 \
  --upload --public -y

2. Base-TN on all optimization / relative-speedup tasks (auto-discovered: algotune-*, gso-speedup-*, plus verifier markers). Each baseline must be correct-but-slow: pass equality / is_solution, fail only the relative speed bar (reward==0, reward_pre_threshold < threshold). A TN that fails correctness (e.g. GSO empty nop → empty_patch) is invalid. Modal amd64; storage ≤10240:

# AlgoTune: naive Task.solve overlay via -a oracle.
# GSO: comment-only no-op overlay via -a oracle (NOT empty nop).
# Prefer k=3 for AlgoTune; k=1 is fine for long GSO tasks.
uv run scripts/run_opt_base_tn.py run \
  --jobs-dir /data/eval-1.3/smoke/opt-base-tn-v2 \
  --env-file /data/harbor-adapters-experiments/.env \
  -e modal --upload
uv run scripts/run_opt_base_tn.py discover   # include list / methods

3. Attest and commit (STOP if either half fails):

uv run scripts/build_push_pin.py oracle-gate \
  --version X.Y --job <oracle-hub-uuid> \
  --base-tn-trials-in /data/eval-1.3/smoke/opt-base-tn-v2/trials.json \
  --base-tn-job <optional-hub-uuid> \
  --digests-in digests.json --execute
git add release-oracle/X.Y.json
git commit -m "release: attest vX.Y oracle gate"
git push origin release/X.Y

--execute requires --job (hub oracle UUID), --base-tn-trials-in from run_opt_base_tn.py (schema opt-base-tn-v2 with per-trial correctness), and --digests-in. Optional --base-tn-job records a Hub UUID. Oracle --trials-in stays dry-run/offline only. Attestation records all-1.0 oracle rewards and correct-but-slow base-TN evidence; alias --execute later requires a hub-linked attestation whose digest_set_sha256 matches the digest map and whose base_tn.requirement is correct-but-slow.

Step 4 — Harbor publish

STOP before --public. Order is mandatory: tasks, then dataset. No --yes on harbor publish — pipe confirmation. Pass tasks/* explicitly (nested dirs are not always auto-collected).

harbor init harbor-index/harbor-index --dataset --output-dir . \
  --description "Harbor Index: 80 agentic … (revision X.Y)." \
  --author "Harbor Framework"
harbor add tasks --scan --to dataset.toml
# confirm 80 unique digested tasks
printf 'y\n' | harbor publish tasks/* --tag X.Y --public --concurrency 20
printf 'y\n' | harbor publish . --tag X.Y --public --concurrency 20

Commit dataset.toml on the release branch before tagging so github-release can attach it. Resolve published dataset content hash from Hub for DATASET_REF.

Step 5 — Leaderboard cutover

1. Create hub leaderboard harbor-index-X-Y (schema in leaderboard/SETUP.md):

harbor hub leaderboard create --config leaderboard.json --json
# or: curl POST …/leaderboard-create with HARBOR_API_KEY (SETUP.md)

2. Update on release branch and open a PR to main (STOP before merge):

Step 6 — Aliases + signed tag + GitHub Release

uv run scripts/build_push_pin.py alias \
  --version X.Y --digests-in digests.json --execute
# STOP — confirm GPG available; prefer signed tags:
git tag -s harbor-index-X.Y -m "harbor-index X.Y (frozen image set)"
git push origin harbor-index-X.Y
gh run list --workflow github-release.yml --limit 3
gh release view harbor-index-X.Y

github-release.yml uses --verify-tag (unsigned tags fail) and attaches release-oracle/X.Y.json + dataset.toml when present. Release notes need an explicit previous harbor-index-* tag (previous_tag_name) because divergent release branches are not linear ancestors (#64).

Hotfix path

1. Land Dockerfile/source fix on main.
2. Cherry-pick onto release/X.Y.
3. Dispatch build-push-pin with mode=hotfix, version X.Y.Z, --ref release/X.Y.
4. Re-run oracle (at least touched required tasks; full gate before alias if attestation must rebind), re-publish if task digests changed, re-alias, tag harbor-index-X.Y.Z if shipping a tagged hotfix.

Drift: after cherry-pick, tree hashes intentionally diverge from old pins. Preflight may use lint --scope release --allow-drift when that flag exists; aggregate still requires drift-free lint --scope release after re-pin. If preflight fails on drift and --allow-drift is missing on the ref, stop and fix tooling before re-dispatching.

Pitfalls

1. Divergent release branches → release-note baseline needs previous tag (#64).
2. alias without --digests-in → --execute refuses.
3. Publish order → tasks before dataset.
4. Evaluate frozen digests / @X.Y, not floating main.
5. Unsigned tags → --verify-tag fails.
6. Hotfix drift / Modal CPU variance can flake oracles (e.g. fastText -march=native).
7. Dirty worktrees → use a clean release worktree.
8. Missing GPG → stop and ask; do not silently fall back without approval.

Done when

• release/X.Y has pins + release-oracle/X.Y.json + dataset.toml
• Hub has public harbor-index/harbor-index@X.Y
• Leaderboard harbor-index-X-Y exists; cutover PR ready/merged on main
• Alias tags stamped; GitHub Release published for harbor-index-X.Y