OpenClaw Release CI
Use this with $release-openclaw-maintainer and $openclaw-testing when a release candidate needs full validation, install/update proof, live provider checks, or CI recovery.
Guardrails
- No version bump, tag, npm publish, GitHub release, or release promotion without explicit operator approval.
- After compaction, resume, or new steering, rewrite the effective goal and
current phase from the latest explicit operator instruction. Do not merge old
scope back into the active release.
- Hold the release scope once a release branch or Code SHA exists. Validate and
ship that exact release; do not turn moving
main into a second work queue.
- Record every active validation run as the immutable tuple Validation SHA +
Tooling SHA + rerun group. Validation SHA maps to the Code SHA for product validation or
the Release SHA for changelog-only validation; it is not a third release
identity. A branch or temporary ref is context and transport.
- Freeze the candidate SHA/ref and Tooling SHA/ref once. Main lineage authorizes
the initial Tooling SHA selection; it does not authorize replacing that
tooling after
main advances.
- Apply a release firebreak after the Code SHA is frozen. Admit only confirmed
product defects, package/provenance defects in the bytes to publish, security
defects, or failures that make publication impossible. Queue other findings
for postpublish confidence or the next beta.
- Use trusted
main workflow revisions as immutable dispatch sources. Do not
adopt newer main code, repair unrelated main CI, wait for broad main health,
or expand a release fix because the workflow source lives on main.
- Once publication binds the Tooling SHA to an exact protected lightweight
release-publish/<12sha>-<provenance-run> tag, that live tag-to-SHA mapping
remains authoritative when main advances. The suffix records tag-creation
provenance; it is not the current parent run id.
- Touch
main only for an operator-requested change or the smallest critical
main-owned blocker that prevents this release and cannot be handled from the
release branch. If the required main landing policy is blocked by unrelated
main failures, report that blocker and keep independent release work moving
instead of healing broader main.
- Validate provider secrets before dispatching expensive full release matrices.
- Linux (
ubuntu) cross-OS lanes gate publication for beta, stable, and full.
Windows/macOS cross-OS lanes run in parallel as advisory coverage. Record
their actual pass/fail conclusions; failures do not block Release Decision,
npm publication, or pnpm release:candidate. Keep normal CI, npm
qualification, Docker, Package Acceptance, performance, and soak gates intact.
- macOS app signing/notarization/appcast and Windows Hub asset promotion run
in parallel with or after npm publication and never delay npm or GitHub
finalization. Their own qualification and artifact gates still apply; track
selected platforms through verified assets and updater evidence separately.
- Do not set GitHub secrets from unvalidated 1Password candidates. If a candidate returns 401/403, leave the existing secret alone and report the exact missing provider.
- Use
$one-password for secret reads/writes: one persistent tmux session, targeted items only, no secret output.
- Watch one parent run plus compact child summaries. Avoid broad
gh run view polling loops; REST quota is easy to burn.
- Fetch logs only for failed or currently-blocking jobs. If quota is low, stop polling and wait for reset.
- Treat live-provider flakes separately from code failures: prove key validity, provider HTTP status, retry evidence, and exact failing lane before editing code.
- A model-list response proves authentication, not billing or inference
entitlement. Mandatory live providers must pass a real completion probe
before release dispatch. Fix the credential first; do not add an alternate
auth path merely to bypass a failed release credential.
- Full Release Validation separates exact-child dispatch, Release Decision,
and Diagnostic Drain. With
fail_fast=false, it makes zero child
cancellation calls; Diagnostic Drain follows every selected child to
terminal unless the collector itself is cancelled or loses GitHub API
access. With
fail_fast=true, Release Decision may cancel only the exact still-active
child that owns a blocking failure.
- Same-parent continuation requires the original root to have been dispatched
with
fail_fast=false. The controller verifies that exact logged input
before any rerun mutation.
- A parent that produced its own sealed candidate artifacts cannot be continued:
GitHub reruns make those prior-attempt artifacts unavailable. Keep the
candidate and Tooling SHAs frozen, supersede that parent, and start a fresh
all-group Full Release Validation.
- After dispatch, one immutable execution-plan artifact records the original
parent attempt, exact child tuples and titles, selected coverage, gates, and
reuse identity. The same bytes are saved under an exact run-ID cache key.
Decision, Drain, manifest writing, evidence validation, and final verification
consume the artifact for their current attempt. A collector retry restores
the cached plan, validates it, re-uploads its artifact, and adopts the same
children; missing plan state is an orchestration failure, not permission to
reconstruct the plan or redispatch.
- Reused evidence is not trusted merely because plan sealing found it. Release
Decision repeats the sealed target SHA, evidence SHA, policy, changed paths,
selected run, root run, source manifest, trusted tooling identity, and
exact-child checks before returning
passed.
- Parent retries select the newest Decision and Drain artifacts independently;
both must bind the same immutable plan even when their source attempts differ.
- Child retries are part of the same immutable plan only when the child run ID,
workflow path, ref, Tooling SHA, dispatch title, event, target, candidate, and
validation inputs remain exact. Newer child attempts replace matching jobs;
jobs absent from a newer attempt carry forward. A duplicate job identity,
missing attempt, regressed attempt, or changed tuple fails closed.
- Use
pnpm frv status|continue --failed|verify for attempt-aware recovery.
The controller is stateless: the immutable execution plan, exact GitHub run
attempts, Diagnostic Drain, and final manifest are the only authorities. It
never writes a tag, package, registry entry, release candidate, or
publication.
- Post-merge controller proof must use the reviewed landed SHA on protected
main through the non-release FRV Proof Broker and FRV Proof Fixture.
Dispatch the broker with the merged pull request number and exact landed
commit. The broker must require that pull request's merge commit to match the
landed commit, prove the landed commit is identical to or an ancestor of its
trusted workflow SHA, and repeat authority, merge, and ancestry checks
immediately before rerunning the fixture.
Require the exact fixed no-op fixture run to advance from its intentional
attempt-one failure to an attempt-two pass. The broker must emit its receipt
without creating a release candidate, release artifact, publication,
repository ref, replacement parent, or other workflow mutation. This is the
hosted GitHub failed-job rerun proof; focused controller tests own immutable
plan eligibility, green-attempt preservation, same-parent collection, and
strict-verifier invocation. Never use a real Full Release Validation run for
this proof. See
Full Release Validation.
- Use one release operator, one transition-only watcher, and at most one
investigator for the current failed surface. Do not build audit-review-plan
trees around a single workflow transition.
- For regular beta/stable releases, Code SHA may already contain final notes
and serve as Release SHA. One successful fresh full parent may qualify both
roles and their exact publication bytes. If notes change afterward, a later
Release SHA may reuse product evidence only when its complete delta from
Code SHA is exactly
CHANGELOG.md; its changed bytes still need qualification.
- Extended-stable validates one exact branch tip; it does not reuse the regular
Code-SHA/Release-SHA evidence model.
- In a sparse worktree or Testbox source sync, first confirm
package.json,
pnpm-lock.yaml, and every source path the selected check reads. If any are
absent, that checkout cannot validate a release dependency or Docker lane:
stop and use the repo remote changed gate or a full task worktree. When the
inputs are present and a release fix changes package.json or
pnpm-lock.yaml, rebuild only the task-owned disposable box with
CI=true pnpm install --frozen-lockfile, then run an explicit
require.resolve() probe before Docker or focused tests. The CI flag permits
pnpm to recreate a prewarmed modules directory without an interactive
confirmation. Do not weaken the lockfile or label sparse-checkout failures
as product/Docker failures.
- If the candidate is rebased or its base SHA changes after warmup, stop the
task-owned box and warm a fresh one before testing. Testbox source sync is
relative to the warmed source tree; continuing can mix an old base file with
a new candidate diff and produce false lockfile or Docker failures.
- Reused Testboxes are provenance-gated after their first successful run.
Source-only edits may reuse the lease; base, dependency, wrapper, or Testbox
workflow drift requires a fresh lease. Do not set
OPENCLAW_TESTBOX_ALLOW_STALE=1 for release evidence.
- For a committed release candidate, warm the box with
blacksmith testbox warmup ... --ref <candidate-branch-or-sha>. Do not rely
on source sync to overlay committed branch changes onto the workflow's
default ref.
Run identity and retry budget
Record Validation SHA, Tooling SHA/ref, target context ref, parent run id,
attempt, and phase before watching or recovering Full Release Validation. Keep
Code SHA and Release SHA as lifecycle roles in the ledger; they may name the
same commit. Record the
immutable Release Publish parent receipt separately from tag provenance.
For the core and plugin npm mutations enforced by this foundation, re-read the
exact protected lightweight tag and revalidate the exact parent run tuple
immediately before each publish or dist-tag mutation. Reject a missing, moved,
annotated, or wrong-SHA tag; a repository, workflow, run id, attempt, tooling
identity, or parent-state mismatch; and any same-name branch. Never refresh
either identity from current main. Treat other privileged writers as blocked
until their dependent enforcement changes land.
Conceptual phases map to current inputs as follows:
beta-publish: release_profile=beta, run_release_soak=false
postpublish-confidence: published package inputs with
run_release_soak=true or explicit focused groups
stable-publish: release_profile=stable
An all run without soak for an actual beta package on its matching canonical
release branch or beta tag records coveragePolicy=npm-beta-v1. It keeps
Linux/macOS/Windows Node, Control UI, plugin, package, install/update,
Linux cross-OS, QA parity, runtime-pair/restart, and tool-coverage gates. Native app
CI, performance, and published-package Telegram are deferred to confidence.
Beta all without soak also defers Package Acceptance Telegram, including
beta-profile checks of main or alpha. Record deferred checks as not run,
never passed. Stable/full, soak, and focused groups retain their coverage;
selected children still require terminal evidence. An absent coverage policy
retains historical full behavior.
Keep at most one active parent for the same Validation SHA + Tooling SHA + rerun
group + release profile + effective soak coverage. Stable/full always include
soak. Distinct coverage profiles can run independently; concurrency does not
cancel an older exact child automatically.
Parent cancellation or timeout leaves adopted identity-checked children
running. The operator must cancel an exact child explicitly when it is no
longer useful. Do not infer a child identity from branch, title prefix, or
latest-run order.
Recover one failed surface with one diagnosis, one fix when needed, and one
narrow retry. Then reassess the release decision. Do not automatically
dispatch rerun_group=all.
For a supported parent, pnpm frv continue --failed --run <parent-run-id>
adopts any active newer child attempt, reruns failed child jobs in parallel,
leaves green children untouched, then reruns the parent once to restore the
immutable plan and seal a trusted all-group manifest. It does not start a
second child retry while an attempt is active. Each child or parent rerun
mutation is sent exactly once; ambiguous transport failures trigger only
bounded read reconciliation. The controller never repeats the mutation, and
provenance drift fails closed.
Inspect without mutation:
pnpm frv status --run <parent-run-id>
pnpm frv verify --run <successful-parent-run-id>
Parents whose immutable plan predates attempt-aware evidence cannot be
continued. Start a fresh all-group Full Release Validation; never reconstruct
old state or dispatch a replacement parent.
Controller retries are ci, plugin-prerelease, install-smoke,
cross-os, live-e2e, package, qa-parity, qa-live, npm-telegram,
or performance. Never use the removed release-checks handle. qa is
only a direct-child manual aggregate, not a controller retry API.
Filtered retries fail closed unless the filter belongs to the selected group.
All-group runs also accept cross_os_suite_filter: for example,
-f cross_os_suite_filter=ubuntu,macos excludes Windows. npm-stable-v1 and
npm-beta-v1 still qualify when advisory OS lanes are omitted, provided all
Linux suites remain selected and the other policy requirements hold.
Never turn an empty derived filter into an unfiltered broad run.
A new all-group parent is justified only when shared orchestration changed,
earlier evidence is invalid for the selected tuple, or the operator explicitly
requests it. Record the invalidating event.
Narrow child or group evidence does not by itself become publish
authorization. Keep it in the evidence ledger for the release owner to judge
against the current publish gate.
Preflight
Before full matrix dispatch, run both pnpm ui:i18n:check and
pnpm native:i18n:check against the frozen trusted target in approved isolation.
Bind both results to that exact SHA; either generated-locale drift blocks
dispatch. Keep target execution outside the trusted dispatch helper—do not
execute an arbitrary target checkout as helper code.
Before expensive full validation, also run pnpm ui:build on the same frozen
trusted target with its frozen dependencies in approved isolation, outside the
trusted dispatch helper. Record the target SHA with the successful production
build, precompressed-asset verification, and startup/largest-asset budget results;
any failure blocks fanout. Do not substitute a dev server or raise budgets to admit
the target.
For local full E2E proof, prepare the frozen, dependency-ready proof checkout
with private QA entries in the initial build:
OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build
Then run the selected E2E command with its normal readiness checks enabled.
scripts/lib/vitest-build-prerequisites.mts requests private QA entries;
scripts/run-node.mts triggers another full build when they are absent. This
preflight avoids rebuilding solely for missing_private_qa_dist. Keep the flag
scoped to this task-owned proof checkout and command. Publication package and
image bytes remain owned by the release workflows and their sealed artifacts.
Before full release validation:
node .agents/skills/release-openclaw-ci/scripts/verify-provider-secrets.mjs --required openai,anthropic,fireworks
gh api rate_limit --jq '.resources.core'
git status --short --branch
git rev-parse HEAD
1Password service-account values are the first source for release provider
preflight. Inject those exact targeted keys first, then run the verifier; use
ambient env only when it was already intentionally injected for this release.
The script prints only provider status and HTTP class, never tokens.
The Anthropic check performs a tiny message completion so exhausted or
non-billable credentials fail before the expensive release matrix.
Dispatch
An early standalone product-performance run is optional beta confidence. If
useful, start it against the frozen Code SHA in parallel with release work:
# Optional early beta confidence; stable/full use the required parent child.
fail_on_regression=false
gh workflow run openclaw-performance.yml \
--repo openclaw/openclaw \
--ref main \
-f target_ref=<code-sha> \
-f profile=release \
-f repeat=3 \
-f deep_profile=false \
-f live_openai_candidate=false \
-f fail_on_regression="$fail_on_regression"
- Do not add a separate mandatory prepublish wait for this optional beta signal.
- Compare available Kova, gateway startup, and CLI startup metrics with earlier
release evidence or clawgrit reports before publish/closeout.
- Call out any regression in the release proof. Treat a major regression as a
release blocker until it is fixed, waived by the operator, or proven to be
infrastructure noise.
- Full Release Validation requires blocking performance evidence for stable
and full profiles.
npm-beta-v1 defers the child; explicit performance
and soak-enabled beta runs retain advisory performance coverage. Every
selected performance child must finish and prove artifact-only publication.
Prefer an immutable trusted-main workflow revision, target the exact Code SHA:
- Keep trusted-workflow checks compatible with frozen release targets. If
main adds a target-owned guard script or package command after the release
branch cut, make the trusted workflow skip only when that target surface is
absent. Repair the smallest trusted-workflow compatibility issue only when it
blocks the release, then rerun validation. Do not port an unrelated runtime
refactor, heal other main failures, or mutate the release candidate just to
satisfy a newer main-only check.
TOOLING_SHA="<exact-main-ancestor-sha>"
node scripts/full-release-validation-at-sha.mjs \
--sha <code-sha> \
--target-ref release/YYYY.M.PATCH \
--workflow-sha "$TOOLING_SHA"
For regular release/* validation, never raw-dispatch the workflow without
target_context_ref (the helper's --target-ref records it). Canonical
release/* and extended-stable/* workflow refs remain supported routes, but
their Telegram child must retain the exact parent workflow ref and SHA through
OIDC and attestation. Trusted-workflow release-branch CI passes target_ref +
release_candidate_ref; never release_gate there — it requires workflow head
== target. (The PR-head ci.yml fallback below is a different dispatch and does
use release_gate=true.)
The release branch may advance after the Code SHA is frozen. The helper accepts
that frozen SHA only while it remains an ancestor of the canonical release
branch and its package version is either the branch's final version or a
matching beta prerelease. Alpha remains on the Tideclaw path with a matching
alpha branch and exact alpha tag. Extended-stable branches and all tags require
an exact package-version match.
Always pass the previously recorded full Tooling SHA for release-branch runs.
Never replace it with a fresh main lookup. The Tooling SHA must declare the
current release-isolation contract; older workflow revisions fail closed.
For immutable workflow proof on a moving main, use
pnpm ci:full-release --sha <code-sha> --target-ref release/YYYY.M.PATCH --workflow-sha <tooling-sha>. Its canonical release-ci/* ref keeps evidence reuse
enabled after proving the workflow commit is still on trusted main lineage.
Pass -f reuse_evidence=false only when the operator intentionally needs a
fresh full run.
If final notes were already committed before fresh full qualification, retain
that Code SHA as Release SHA and use the same successful parent/attempt and
its exact prepared bytes for candidate and publication checks. Required gates,
final channel-specific SDK review and acknowledgement still apply.
Only if notes change after qualification, commit exactly CHANGELOG.md and
optionally run the helper against the new Release SHA with reuse. That parent must report
policy=changelog-only-release-v1, evidenceSha=<code-sha>, and
changedPaths=["CHANGELOG.md"]; it should reuse the product matrix instead of
dispatching child lanes. Npm preflight and package/install acceptance still run
against the exact Release SHA and its new tarball bytes.
Current all-group FRV also owns read-only npm source/build/qualification and
Docker preparation. Use its successful run as preflight_run_id; the candidate
helper defaults to that run. Do not dispatch a second npm preflight unless
recovering historical separate evidence. Regular final qualification records
SDK reports for both beta and latest; review the acknowledgement for the
actual publication channel. Prepared descriptors live in publicationArtifacts in
the exact final manifest. Product evidence reuse never substitutes Code-SHA
package or image bytes for the final Release SHA. A parent that produced these
artifacts needs a fresh all-group FRV instead of same-parent continuation.
The SHA-pinned helper infers beta for matching beta release candidates and
exact alpha tags, and stable for stable/correction versions, then passes the
Validation SHA + Tooling SHA run identity. Canonical beta all without soak
uses npm-beta-v1; main, alpha, and non-beta targets do not qualify for that
policy. Run deferred native, performance, Telegram, broad live QA, and E2E as
postpublish confidence with the exact published package and
run_release_soak=true or explicit groups. Stable and full profiles force the
release soak. Native artifact publication still requires its own build,
signing, notarization, and promotion gates. Use a narrow rerun_group after
focused fixes; never widen automatically.
Publish with openclaw-release-publish.yml using release_profile=from-validation
unless a maintainer intentionally wants to cross-check a specific profile; the
publish workflow reads the effective profile from the full-validation manifest.
Extended-stable validation
Use one remote-only procedure for .33+ extended-stable validation. Keep these
four identities separate:
- Validation SHA: exact 40-character candidate commit to validate.
- Tooling SHA: exact trusted-main commit whose workflows and helpers run.
- Context ref: canonical
extended-stable/YYYY.M.33 branch containing the
candidate.
- Workflow transport ref: immutable
release-ci/<tooling-sha-prefix>-<unique-id> branch at the Tooling SHA.
GitHub workflow dispatch --ref accepts a branch or tag name, not a raw commit
SHA. Never raw-dispatch this validation or hand-assemble its identity inputs.
Use the checked helper exclusively:
VALIDATION_SHA="<exact-candidate-sha>"
TOOLING_SHA="<recorded-full-main-ancestor-sha>"
CONTEXT_REF="extended-stable/YYYY.M.33"
pnpm ci:full-release \
--sha "$VALIDATION_SHA" \
--target-ref "$CONTEXT_REF" \
--workflow-sha "$TOOLING_SHA" \
-f release_profile=stable \
-f run_release_soak=true \
-f fail_fast=false \
-f rerun_group=all \
-f reuse_evidence=false \
-f dispatch_release_evidence=false
The helper verifies both SHAs, creates the transport ref with the equivalent of
the following GitHub refs operation, and dispatches from that branch:
gh api --method POST repos/openclaw/openclaw/git/refs \
-f ref="refs/heads/release-ci/${TOOLING_SHA:0:12}-<unique-id>" \
-f sha="$TOOLING_SHA"
Do not run that operation separately. The helper also supplies
ref=$VALIDATION_SHA, expected_sha=$VALIDATION_SHA,
target_context_ref=$CONTEXT_REF, and this exact trusted identity:
{"fullRef":"refs/heads/main","ref":"main","sha":"<tooling-sha>"}
Outside this extended-stable procedure, a direct canonical-branch dispatch is
valid only when that branch's own head is both the Validation SHA and the
trusted workflow implementation to execute. It cannot use a different
trusted-main Tooling SHA. Current extended-stable validation requires distinct
trusted-main tooling, so it must use the immutable release-ci/* transport
above. Direct canonical-branch and mutable-main dispatches are not valid
alternatives for this procedure.
Accept only a complete rerun_group=all run with a supported exact-target
manifest. Bind its workflow SHA separately from the candidate SHA; require the
manifest target, package versions, saved run_attempt, and final tag to identify
the same candidate. Reject narrow runs, untrusted tooling, mismatched targets,
and earlier-attempt evidence.
Run the npm preflight separately from trusted main. Here tag is the exact
candidate SHA; it is an npm-preflight input, not the workflow transport ref:
gh workflow run openclaw-npm-release.yml \
--repo openclaw/openclaw \
--ref main \
-f tag="$VALIDATION_SHA" \
-f preflight_only=true \
-f npm_dist_tag=extended-stable \
-f release_candidate_branch="$CONTEXT_REF"
This standalone run is a supplemental validation-only preflight. Do not pass
its run ID as publication preflight_run_id: a main workflow head does not
have the canonical candidate branch/SHA identity required by that publication
input. Publication continues to use the Full Release Validation run's
manifest-bound integrated npm artifact and exact run attempt.
Product failures need an approved backport. Frozen-target tooling failures need
the smallest behavior-preserving repair. Provider, approval, runner, or log
races keep the candidate unchanged. Record repairs and superseded runs; any
branch change requires a new complete parent. Omit only an explicitly
unsupported frozen-target scenario, never a required behavior or package.
Watch
Use the transition-only summary watcher instead of repeated raw polling:
node scripts/release-ci-summary.mjs <full-release-run-id> --watch
Do not start this watcher when the SHA-pinned helper is still the foreground
owner. The helper reads the exact Release Decision artifact itself. On
blocked_diagnostics_running, it exits nonzero immediately, keeps the temporary
refs, and leaves Diagnostic Drain collecting the remaining terminal evidence.
The watcher behaves the same way for separately dispatched parents: it reports
the Release Decision blocker once and exits while the drain continues.
For a one-shot snapshot:
node scripts/release-ci-summary.mjs <full-release-run-id>
release-ci-summary accepts Full Release Validation parent runs only.
Diverged release-branch logs: --first-parent plus a bounded count.
Stop watchers before ending the turn or switching strategy.
Interpret state precisely:
qualifying: no decisive blocker yet; selected children are still active.
blocked_diagnostics_running: publication is blocked; Diagnostic Drain is
still collecting independent failures. Diagnose now, but do not retry until
the drain is terminal.
passed: all required policy and exact-child evidence passed.
blocked_complete: publication is blocked and all selected diagnostics are
terminal.
orchestration_error: GitHub API or collector failure prevented a verdict.
This is not a provenance mismatch. Recover the collector against the same
exact children; never redispatch tests to repair collection.
cancelled_with_children: the collector was cancelled while exact children
remained active.
Read advisory entries separately from Release Decision. Windows/macOS
cross-OS lanes retain their actual conclusions in the manifest and summary;
passed does not mean those advisory lanes passed. Selected lanes still need
terminal evidence, and filtered-out lanes are not run, never passed.
The full-release-diagnostics-<run-id>-<attempt> artifact is the terminal
failure and timing manifest. Use it after an early blocker instead of
restarting all merely to discover what the still-running children found.
The stable full-release-execution-plan-<run-id> artifact is the identity
source within each collector attempt; retry attempts restore its immutable
run-ID-cached bytes first.
Failure Triage
- Confirm parent SHA and child run IDs.
- List failed jobs only:
gh run view <child-run-id> --repo openclaw/openclaw --json jobs \
--jq '.jobs[] | select(.conclusion=="failure" or .conclusion=="timed_out" or .conclusion=="cancelled") | [.databaseId,.name,.conclusion,.url] | @tsv'
- Fetch one failed job log. If rate-limited, note reset time and avoid more REST calls.
- For secret-looking failures, validate a real completion from the same secret source before editing code. A successful model-list request is insufficient.
Claude CLI subscription credentials are a separate native auth path; prove
them in a clean-home CLI probe, never as a substitute for a required
Anthropic API-key lane.
- For live-cache failures, inspect whether it is missing/invalid key, empty text, provider refusal, timeout, or baseline miss. Do not weaken release gates without clear provider evidence.
- Classify before editing:
- confirmed product/code failure: fix the release branch, freeze a new Code
SHA, and invalidate product evidence
- harness/tooling/provenance failure: keep the Code SHA, fix the smallest
owning surface, and retry only the failed surface with the required Tooling
SHA
- infrastructure/credential failure: keep both SHAs, repair the external
prerequisite, and retry only the failed surface
- wrapper/monitor failure: keep the child and candidate identities; record
the wrapper result separately from the child result
- changelog/release-note failure: change only
CHANGELOG.md, keep Code SHA
evidence, and repeat Release SHA proof
- publish child/registry selector failure: keep Release SHA and resume the
failed child; never rebuild an immutable version that already published
Only the first class changes the Code SHA. After one diagnosis/fix/narrow
retry, reassess instead of starting another all-group cycle.
- If a required PR CI run is capacity-stalled with queued jobs and no active
jobs, do not cancel unrelated work or accept a generic manual dispatch.
First verify the PR head carries the current fallback schema:
gh api 'repos/openclaw/openclaw/contents/.github/workflows/ci.yml?ref=<pr-head-branch>' --jq .content | base64 --decode | rg -q 'pull_request_number:'. If absent,
refresh the PR head from main and use the new head SHA; let normal CI run
before considering another fallback.
From the PR head branch, dispatch the explicit exact-SHA fallback:
gh workflow run ci.yml --repo openclaw/openclaw --ref <pr-head-branch> -f target_ref=<full-pr-sha> -f pull_request_number=<pr-number> -f include_android=true -f release_gate=true.
It runs on GitHub-hosted runners and is accepted only when its run title is
CI release gate <full-pr-sha>. Record the stalled Blacksmith run and the
fallback run in release evidence.
If Blacksmith Build Artifacts Testbox is the only remaining required gate
and remains queued without a runner, that completed exact fallback may cover
it because CI's build-artifacts job already builds, packages, and smoke
tests the artifacts. Do not use this coverage after the artifact workflow
starts or completes non-successfully.
Evidence
Record:
- release lifecycle ledger: Code SHA, Release SHA, and Tooling SHA for regular
releases; canonical branch, exact SHA, and immutable tag for extended-stable
- evidence-reuse policy, coverage policy, and complete changed-path set
- active full parent run URL, attempt, workflow SHA, and any superseded parent
with the exact replacement reason
- selected child run IDs and conclusions: CI, Release Checks, Plugin Prerelease, NPM Telegram, Product Performance; record deferred confidence as not run
- Windows/macOS cross-OS advisory lane classifications and actual conclusions
- performance comparison result versus earlier releases when available
- targeted local proof commands
- provider-secret preflight result
- frozen-target compatibility repairs or omitted inapplicable scenarios, with
their source PRs and invariant
- known gaps or unrelated failures
For lessons and recovery patterns, read references/release-ci-notes.md.
1---2name: release-openclaw-ci3description: Run, watch, debug, and summarize OpenClaw full release CI, release checks, live provider gates, install/update proofs, and release-secret preflights.4---5
6# OpenClaw Release CI
7
8Use this with `$release-openclaw-maintainer` and `$openclaw-testing` when a release candidate needs full validation, install/update proof, live provider checks, or CI recovery.
9
10## Guardrails
11
12- No version bump, tag, npm publish, GitHub release, or release promotion without explicit operator approval.
13- After compaction, resume, or new steering, rewrite the effective goal and
14 current phase from the latest explicit operator instruction. Do not merge old
15 scope back into the active release.
16- Hold the release scope once a release branch or Code SHA exists. Validate and
17 ship that exact release; do not turn moving `main` into a second work queue.
18- Record every active validation run as the immutable tuple **Validation SHA +
19 Tooling SHA + rerun group**. Validation SHA maps to the Code SHA for product validation or
20 the Release SHA for changelog-only validation; it is not a third release
21 identity. A branch or temporary ref is context and transport.
22- Freeze the candidate SHA/ref and Tooling SHA/ref once. Main lineage authorizes
23 the initial Tooling SHA selection; it does not authorize replacing that
24 tooling after `main` advances.
25- Apply a release firebreak after the Code SHA is frozen. Admit only confirmed
26 product defects, package/provenance defects in the bytes to publish, security
27 defects, or failures that make publication impossible. Queue other findings
28 for postpublish confidence or the next beta.
29- Use trusted `main` workflow revisions as immutable dispatch sources. Do not
30 adopt newer main code, repair unrelated main CI, wait for broad main health,
31 or expand a release fix because the workflow source lives on `main`.
32- Once publication binds the Tooling SHA to an exact protected lightweight
33 `release-publish/<12sha>-<provenance-run>` tag, that live tag-to-SHA mapping
34 remains authoritative when `main` advances. The suffix records tag-creation
35 provenance; it is not the current parent run id.
36- Touch `main` only for an operator-requested change or the smallest critical
37 main-owned blocker that prevents this release and cannot be handled from the
38 release branch. If the required main landing policy is blocked by unrelated
39 main failures, report that blocker and keep independent release work moving
40 instead of healing broader main.
41- Validate provider secrets before dispatching expensive full release matrices.
42- Linux (`ubuntu`) cross-OS lanes gate publication for beta, stable, and full.
43 Windows/macOS cross-OS lanes run in parallel as advisory coverage. Record
44 their actual pass/fail conclusions; failures do not block Release Decision,
45 npm publication, or `pnpm release:candidate`. Keep normal CI, npm
46 qualification, Docker, Package Acceptance, performance, and soak gates intact.
47- macOS app signing/notarization/appcast and Windows Hub asset promotion run
48 in parallel with or after npm publication and never delay npm or GitHub
49 finalization. Their own qualification and artifact gates still apply; track
50 selected platforms through verified assets and updater evidence separately.
51- Do not set GitHub secrets from unvalidated 1Password candidates. If a candidate returns 401/403, leave the existing secret alone and report the exact missing provider.
52- Use `$one-password` for secret reads/writes: one persistent tmux session, targeted items only, no secret output.
53- Watch one parent run plus compact child summaries. Avoid broad `gh run view` polling loops; REST quota is easy to burn.
54- Fetch logs only for failed or currently-blocking jobs. If quota is low, stop polling and wait for reset.
55- Treat live-provider flakes separately from code failures: prove key validity, provider HTTP status, retry evidence, and exact failing lane before editing code.
56- A model-list response proves authentication, not billing or inference
57 entitlement. Mandatory live providers must pass a real completion probe
58 before release dispatch. Fix the credential first; do not add an alternate
59 auth path merely to bypass a failed release credential.
60- Full Release Validation separates exact-child dispatch, Release Decision,
61 and Diagnostic Drain. With `fail_fast=false`, it makes zero child
62 cancellation calls; Diagnostic Drain follows every selected child to
63 terminal unless the collector itself is cancelled or loses GitHub API
64 access. With
65 `fail_fast=true`, Release Decision may cancel only the exact still-active
66 child that owns a blocking failure.
67- Same-parent continuation requires the original root to have been dispatched
68 with `fail_fast=false`. The controller verifies that exact logged input
69 before any rerun mutation.
70- A parent that produced its own sealed candidate artifacts cannot be continued:
71 GitHub reruns make those prior-attempt artifacts unavailable. Keep the
72 candidate and Tooling SHAs frozen, supersede that parent, and start a fresh
73 all-group Full Release Validation.
74- After dispatch, one immutable execution-plan artifact records the original
75 parent attempt, exact child tuples and titles, selected coverage, gates, and
76 reuse identity. The same bytes are saved under an exact run-ID cache key.
77 Decision, Drain, manifest writing, evidence validation, and final verification
78 consume the artifact for their current attempt. A collector retry restores
79 the cached plan, validates it, re-uploads its artifact, and adopts the same
80 children; missing plan state is an orchestration failure, not permission to
81 reconstruct the plan or redispatch.
82- Reused evidence is not trusted merely because plan sealing found it. Release
83 Decision repeats the sealed target SHA, evidence SHA, policy, changed paths,
84 selected run, root run, source manifest, trusted tooling identity, and
85 exact-child checks before returning `passed`.
86- Parent retries select the newest Decision and Drain artifacts independently;
87 both must bind the same immutable plan even when their source attempts differ.
88- Child retries are part of the same immutable plan only when the child run ID,
89 workflow path, ref, Tooling SHA, dispatch title, event, target, candidate, and
90 validation inputs remain exact. Newer child attempts replace matching jobs;
91 jobs absent from a newer attempt carry forward. A duplicate job identity,
92 missing attempt, regressed attempt, or changed tuple fails closed.
93- Use `pnpm frv status|continue --failed|verify` for attempt-aware recovery.
94 The controller is stateless: the immutable execution plan, exact GitHub run
95 attempts, Diagnostic Drain, and final manifest are the only authorities. It
96 never writes a tag, package, registry entry, release candidate, or
97 publication.
98- Post-merge controller proof must use the reviewed landed SHA on protected
99 `main` through the non-release `FRV Proof Broker` and `FRV Proof Fixture`.
100 Dispatch the broker with the merged pull request number and exact landed
101 commit. The broker must require that pull request's merge commit to match the
102 landed commit, prove the landed commit is identical to or an ancestor of its
103 trusted workflow SHA, and repeat authority, merge, and ancestry checks
104 immediately before rerunning the fixture.
105 Require the exact fixed no-op fixture run to advance from its intentional
106 attempt-one failure to an attempt-two pass. The broker must emit its receipt
107 without creating a release candidate, release artifact, publication,
108 repository ref, replacement parent, or other workflow mutation. This is the
109 hosted GitHub failed-job rerun proof; focused controller tests own immutable
110 plan eligibility, green-attempt preservation, same-parent collection, and
111 strict-verifier invocation. Never use a real Full Release Validation run for
112 this proof. See
113 [Full Release Validation](/reference/full-release-validation#post-merge-continuation-proof).
114- Use one release operator, one transition-only watcher, and at most one
115 investigator for the current failed surface. Do not build audit-review-plan
116 trees around a single workflow transition.
117- For regular beta/stable releases, Code SHA may already contain final notes
118 and serve as Release SHA. One successful fresh full parent may qualify both
119 roles and their exact publication bytes. If notes change afterward, a later
120 Release SHA may reuse product evidence only when its complete delta from
121 Code SHA is exactly `CHANGELOG.md`; its changed bytes still need qualification.
122- Extended-stable validates one exact branch tip; it does not reuse the regular
123 Code-SHA/Release-SHA evidence model.
124- In a sparse worktree or Testbox source sync, first confirm `package.json`,
125 `pnpm-lock.yaml`, and every source path the selected check reads. If any are
126 absent, that checkout cannot validate a release dependency or Docker lane:
127 stop and use the repo remote changed gate or a full task worktree. When the
128 inputs are present and a release fix changes `package.json` or
129 `pnpm-lock.yaml`, rebuild only the task-owned disposable box with
130 `CI=true pnpm install --frozen-lockfile`, then run an explicit
131 `require.resolve()` probe before Docker or focused tests. The CI flag permits
132 pnpm to recreate a prewarmed modules directory without an interactive
133 confirmation. Do not weaken the lockfile or label sparse-checkout failures
134 as product/Docker failures.
135- If the candidate is rebased or its base SHA changes after warmup, stop the
136 task-owned box and warm a fresh one before testing. Testbox source sync is
137 relative to the warmed source tree; continuing can mix an old base file with
138 a new candidate diff and produce false lockfile or Docker failures.
139- Reused Testboxes are provenance-gated after their first successful run.
140 Source-only edits may reuse the lease; base, dependency, wrapper, or Testbox
141 workflow drift requires a fresh lease. Do not set
142 `OPENCLAW_TESTBOX_ALLOW_STALE=1` for release evidence.
143- For a committed release candidate, warm the box with
144 `blacksmith testbox warmup ... --ref <candidate-branch-or-sha>`. Do not rely
145 on source sync to overlay committed branch changes onto the workflow's
146 default ref.
147
148## Run identity and retry budget
149
150Record Validation SHA, Tooling SHA/ref, target context ref, parent run id,
151attempt, and phase before watching or recovering Full Release Validation. Keep
152Code SHA and Release SHA as lifecycle roles in the ledger; they may name the
153same commit. Record the
154immutable Release Publish parent receipt separately from tag provenance.
155
156For the core and plugin npm mutations enforced by this foundation, re-read the
157exact protected lightweight tag and revalidate the exact parent run tuple
158immediately before each publish or dist-tag mutation. Reject a missing, moved,
159annotated, or wrong-SHA tag; a repository, workflow, run id, attempt, tooling
160identity, or parent-state mismatch; and any same-name branch. Never refresh
161either identity from current `main`. Treat other privileged writers as blocked
162until their dependent enforcement changes land.
163
164- Conceptual phases map to current inputs as follows:
165 - `beta-publish`: `release_profile=beta`, `run_release_soak=false`
166 - `postpublish-confidence`: published package inputs with
167 `run_release_soak=true` or explicit focused groups
168 - `stable-publish`: `release_profile=stable`
169- An `all` run without soak for an actual beta package on its matching canonical
170 release branch or beta tag records `coveragePolicy=npm-beta-v1`. It keeps
171 Linux/macOS/Windows Node, Control UI, plugin, package, install/update,
172 Linux cross-OS, QA parity, runtime-pair/restart, and tool-coverage gates. Native app
173 CI, performance, and published-package Telegram are deferred to confidence.
174 Beta `all` without soak also defers Package Acceptance Telegram, including
175 beta-profile checks of `main` or alpha. Record deferred checks as not run,
176 never passed. Stable/full, soak, and focused groups retain their coverage;
177 selected children still require terminal evidence. An absent coverage policy
178 retains historical full behavior.
179- Keep at most one active parent for the same Validation SHA + Tooling SHA + rerun
180 group + release profile + effective soak coverage. Stable/full always include
181 soak. Distinct coverage profiles can run independently; concurrency does not
182 cancel an older exact child automatically.
183- Parent cancellation or timeout leaves adopted identity-checked children
184 running. The operator must cancel an exact child explicitly when it is no
185 longer useful. Do not infer a child identity from branch, title prefix, or
186 latest-run order.
187- Recover one failed surface with one diagnosis, one fix when needed, and one
188 narrow retry. Then reassess the release decision. Do not automatically
189 dispatch `rerun_group=all`.
190- For a supported parent, `pnpm frv continue --failed --run <parent-run-id>`
191 adopts any active newer child attempt, reruns failed child jobs in parallel,
192 leaves green children untouched, then reruns the parent once to restore the
193 immutable plan and seal a trusted all-group manifest. It does not start a
194 second child retry while an attempt is active. Each child or parent rerun
195 mutation is sent exactly once; ambiguous transport failures trigger only
196 bounded read reconciliation. The controller never repeats the mutation, and
197 provenance drift fails closed.
198- Inspect without mutation:
199
200 ```bash
201 pnpm frv status --run <parent-run-id>
202 pnpm frv verify --run <successful-parent-run-id>
203 ```
204
205- Parents whose immutable plan predates attempt-aware evidence cannot be
206 continued. Start a fresh all-group Full Release Validation; never reconstruct
207 old state or dispatch a replacement parent.
208
209- Controller retries are `ci`, `plugin-prerelease`, `install-smoke`,
210 `cross-os`, `live-e2e`, `package`, `qa-parity`, `qa-live`, `npm-telegram`,
211 or `performance`. Never use the removed `release-checks` handle. `qa` is
212 only a direct-child manual aggregate, not a controller retry API.
213- Filtered retries fail closed unless the filter belongs to the selected group.
214 All-group runs also accept `cross_os_suite_filter`: for example,
215 `-f cross_os_suite_filter=ubuntu,macos` excludes Windows. `npm-stable-v1` and
216 `npm-beta-v1` still qualify when advisory OS lanes are omitted, provided all
217 Linux suites remain selected and the other policy requirements hold.
218 Never turn an empty derived filter into an unfiltered broad run.
219- A new all-group parent is justified only when shared orchestration changed,
220 earlier evidence is invalid for the selected tuple, or the operator explicitly
221 requests it. Record the invalidating event.
222- Narrow child or group evidence does not by itself become publish
223 authorization. Keep it in the evidence ledger for the release owner to judge
224 against the current publish gate.
225
226## Preflight
227
228Before full matrix dispatch, run both `pnpm ui:i18n:check` and
229`pnpm native:i18n:check` against the frozen trusted target in approved isolation.
230Bind both results to that exact SHA; either generated-locale drift blocks
231dispatch. Keep target execution outside the trusted dispatch helper—do not
232execute an arbitrary target checkout as helper code.
233
234Before expensive full validation, also run `pnpm ui:build` on the same frozen
235trusted target with its frozen dependencies in approved isolation, outside the
236trusted dispatch helper. Record the target SHA with the successful production
237build, precompressed-asset verification, and startup/largest-asset budget results;
238any failure blocks fanout. Do not substitute a dev server or raise budgets to admit
239the target.
240
241For local full E2E proof, prepare the frozen, dependency-ready proof checkout
242with private QA entries in the initial build:
243
244```bash
245OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build
246```
247
248Then run the selected E2E command with its normal readiness checks enabled.
249`scripts/lib/vitest-build-prerequisites.mts` requests private QA entries;
250`scripts/run-node.mts` triggers another full build when they are absent. This
251preflight avoids rebuilding solely for `missing_private_qa_dist`. Keep the flag
252scoped to this task-owned proof checkout and command. Publication package and
253image bytes remain owned by the release workflows and their sealed artifacts.
254
255Before full release validation:
256
257```bash
258node .agents/skills/release-openclaw-ci/scripts/verify-provider-secrets.mjs --required openai,anthropic,fireworks
259gh api rate_limit --jq '.resources.core'
260git status --short --branch
261git rev-parse HEAD
262```
263
2641Password service-account values are the first source for release provider
265preflight. Inject those exact targeted keys first, then run the verifier; use
266ambient env only when it was already intentionally injected for this release.
267The script prints only provider status and HTTP class, never tokens.
268The Anthropic check performs a tiny message completion so exhausted or
269non-billable credentials fail before the expensive release matrix.
270
271## Dispatch
272
273An early standalone product-performance run is optional beta confidence. If
274useful, start it against the frozen Code SHA in parallel with release work:
275
276```bash
277# Optional early beta confidence; stable/full use the required parent child.
278fail_on_regression=false
279gh workflow run openclaw-performance.yml \
280 --repo openclaw/openclaw \
281 --ref main \
282 -f target_ref=<code-sha> \
283 -f profile=release \
284 -f repeat=3 \
285 -f deep_profile=false \
286 -f live_openai_candidate=false \
287 -f fail_on_regression="$fail_on_regression"
288```
289
290- Do not add a separate mandatory prepublish wait for this optional beta signal.
291- Compare available Kova, gateway startup, and CLI startup metrics with earlier
292 release evidence or clawgrit reports before publish/closeout.
293- Call out any regression in the release proof. Treat a major regression as a
294 release blocker until it is fixed, waived by the operator, or proven to be
295 infrastructure noise.
296- Full Release Validation requires blocking performance evidence for stable
297 and full profiles. `npm-beta-v1` defers the child; explicit `performance`
298 and soak-enabled beta runs retain advisory performance coverage. Every
299 selected performance child must finish and prove artifact-only publication.
300
301Prefer an immutable trusted-main workflow revision, target the exact Code SHA:
302
303- Keep trusted-workflow checks compatible with frozen release targets. If
304 `main` adds a target-owned guard script or package command after the release
305 branch cut, make the trusted workflow skip only when that target surface is
306 absent. Repair the smallest trusted-workflow compatibility issue only when it
307 blocks the release, then rerun validation. Do not port an unrelated runtime
308 refactor, heal other main failures, or mutate the release candidate just to
309 satisfy a newer `main`-only check.
310
311```bash
312TOOLING_SHA="<exact-main-ancestor-sha>"
313node scripts/full-release-validation-at-sha.mjs \
314 --sha <code-sha> \
315 --target-ref release/YYYY.M.PATCH \
316 --workflow-sha "$TOOLING_SHA"
317```
318
319For regular `release/*` validation, never raw-dispatch the workflow without
320`target_context_ref` (the helper's `--target-ref` records it). Canonical
321`release/*` and `extended-stable/*` workflow refs remain supported routes, but
322their Telegram child must retain the exact parent workflow ref and SHA through
323OIDC and attestation. Trusted-workflow release-branch CI passes `target_ref` +
324`release_candidate_ref`; never `release_gate` there — it requires workflow head
325== target. (The PR-head ci.yml fallback below is a different dispatch and does
326use `release_gate=true`.)
327
328The release branch may advance after the Code SHA is frozen. The helper accepts
329that frozen SHA only while it remains an ancestor of the canonical release
330branch and its package version is either the branch's final version or a
331matching beta prerelease. Alpha remains on the Tideclaw path with a matching
332alpha branch and exact alpha tag. Extended-stable branches and all tags require
333an exact package-version match.
334Always pass the previously recorded full Tooling SHA for release-branch runs.
335Never replace it with a fresh `main` lookup. The Tooling SHA must declare the
336current release-isolation contract; older workflow revisions fail closed.
337
338For immutable workflow proof on a moving `main`, use
339`pnpm ci:full-release --sha <code-sha> --target-ref
340release/YYYY.M.PATCH --workflow-sha <tooling-sha>`. Its canonical `release-ci/*` ref keeps evidence reuse
341enabled after proving the workflow commit is still on trusted `main` lineage.
342Pass `-f reuse_evidence=false` only when the operator intentionally needs a
343fresh full run.
344
345If final notes were already committed before fresh full qualification, retain
346that Code SHA as Release SHA and use the same successful parent/attempt and
347its exact prepared bytes for candidate and publication checks. Required gates,
348final channel-specific SDK review and acknowledgement still apply.
349
350Only if notes change after qualification, commit exactly `CHANGELOG.md` and
351optionally run the helper against the new Release SHA with reuse. That parent must report
352`policy=changelog-only-release-v1`, `evidenceSha=<code-sha>`, and
353`changedPaths=["CHANGELOG.md"]`; it should reuse the product matrix instead of
354dispatching child lanes. Npm preflight and package/install acceptance still run
355against the exact Release SHA and its new tarball bytes.
356
357Current all-group FRV also owns read-only npm source/build/qualification and
358Docker preparation. Use its successful run as `preflight_run_id`; the candidate
359helper defaults to that run. Do not dispatch a second npm preflight unless
360recovering historical separate evidence. Regular final qualification records
361SDK reports for both `beta` and `latest`; review the acknowledgement for the
362actual publication channel. Prepared descriptors live in `publicationArtifacts` in
363the exact final manifest. Product evidence reuse never substitutes Code-SHA
364package or image bytes for the final Release SHA. A parent that produced these
365artifacts needs a fresh all-group FRV instead of same-parent continuation.
366
367The SHA-pinned helper infers `beta` for matching beta release candidates and
368exact alpha tags, and `stable` for stable/correction versions, then passes the
369Validation SHA + Tooling SHA run identity. Canonical beta `all` without soak
370uses `npm-beta-v1`; `main`, alpha, and non-beta targets do not qualify for that
371policy. Run deferred native, performance, Telegram, broad live QA, and E2E as
372postpublish confidence with the exact published package and
373`run_release_soak=true` or explicit groups. Stable and full profiles force the
374release soak. Native artifact publication still requires its own build,
375signing, notarization, and promotion gates. Use a narrow `rerun_group` after
376focused fixes; never widen automatically.
377Publish with `openclaw-release-publish.yml` using `release_profile=from-validation`
378unless a maintainer intentionally wants to cross-check a specific profile; the
379publish workflow reads the effective profile from the full-validation manifest.
380
381### Extended-stable validation
382
383Use one remote-only procedure for `.33+` extended-stable validation. Keep these
384four identities separate:
385
386- **Validation SHA:** exact 40-character candidate commit to validate.
387- **Tooling SHA:** exact trusted-main commit whose workflows and helpers run.
388- **Context ref:** canonical `extended-stable/YYYY.M.33` branch containing the
389 candidate.
390- **Workflow transport ref:** immutable
391 `release-ci/<tooling-sha-prefix>-<unique-id>` branch at the Tooling SHA.
392
393GitHub workflow dispatch `--ref` accepts a branch or tag name, not a raw commit
394SHA. Never raw-dispatch this validation or hand-assemble its identity inputs.
395Use the checked helper exclusively:
396
397```bash
398VALIDATION_SHA="<exact-candidate-sha>"
399TOOLING_SHA="<recorded-full-main-ancestor-sha>"
400CONTEXT_REF="extended-stable/YYYY.M.33"
401pnpm ci:full-release \
402 --sha "$VALIDATION_SHA" \
403 --target-ref "$CONTEXT_REF" \
404 --workflow-sha "$TOOLING_SHA" \
405 -f release_profile=stable \
406 -f run_release_soak=true \
407 -f fail_fast=false \
408 -f rerun_group=all \
409 -f reuse_evidence=false \
410 -f dispatch_release_evidence=false
411```
412
413The helper verifies both SHAs, creates the transport ref with the equivalent of
414the following GitHub refs operation, and dispatches from that branch:
415
416```bash
417gh api --method POST repos/openclaw/openclaw/git/refs \
418 -f ref="refs/heads/release-ci/${TOOLING_SHA:0:12}-<unique-id>" \
419 -f sha="$TOOLING_SHA"
420```
421
422Do not run that operation separately. The helper also supplies
423`ref=$VALIDATION_SHA`, `expected_sha=$VALIDATION_SHA`,
424`target_context_ref=$CONTEXT_REF`, and this exact trusted identity:
425
426```text
427{"fullRef":"refs/heads/main","ref":"main","sha":"<tooling-sha>"}
428```
429
430Outside this extended-stable procedure, a direct canonical-branch dispatch is
431valid only when that branch's own head is both the Validation SHA and the
432trusted workflow implementation to execute. It cannot use a different
433trusted-main Tooling SHA. Current extended-stable validation requires distinct
434trusted-main tooling, so it must use the immutable `release-ci/*` transport
435above. Direct canonical-branch and mutable-`main` dispatches are not valid
436alternatives for this procedure.
437
438Accept only a complete `rerun_group=all` run with a supported exact-target
439manifest. Bind its workflow SHA separately from the candidate SHA; require the
440manifest target, package versions, saved `run_attempt`, and final tag to identify
441the same candidate. Reject narrow runs, untrusted tooling, mismatched targets,
442and earlier-attempt evidence.
443
444Run the npm preflight separately from trusted `main`. Here `tag` is the exact
445candidate SHA; it is an npm-preflight input, not the workflow transport ref:
446
447```bash
448gh workflow run openclaw-npm-release.yml \
449 --repo openclaw/openclaw \
450 --ref main \
451 -f tag="$VALIDATION_SHA" \
452 -f preflight_only=true \
453 -f npm_dist_tag=extended-stable \
454 -f release_candidate_branch="$CONTEXT_REF"
455```
456
457This standalone run is a supplemental validation-only preflight. Do not pass
458its run ID as publication `preflight_run_id`: a `main` workflow head does not
459have the canonical candidate branch/SHA identity required by that publication
460input. Publication continues to use the Full Release Validation run's
461manifest-bound integrated npm artifact and exact run attempt.
462
463Product failures need an approved backport. Frozen-target tooling failures need
464the smallest behavior-preserving repair. Provider, approval, runner, or log
465races keep the candidate unchanged. Record repairs and superseded runs; any
466branch change requires a new complete parent. Omit only an explicitly
467unsupported frozen-target scenario, never a required behavior or package.
468
469## Watch
470
471Use the transition-only summary watcher instead of repeated raw polling:
472
473```bash
474node scripts/release-ci-summary.mjs <full-release-run-id> --watch
475```
476
477Do not start this watcher when the SHA-pinned helper is still the foreground
478owner. The helper reads the exact Release Decision artifact itself. On
479`blocked_diagnostics_running`, it exits nonzero immediately, keeps the temporary
480refs, and leaves Diagnostic Drain collecting the remaining terminal evidence.
481The watcher behaves the same way for separately dispatched parents: it reports
482the Release Decision blocker once and exits while the drain continues.
483
484For a one-shot snapshot:
485
486```bash
487node scripts/release-ci-summary.mjs <full-release-run-id>
488```
489
490`release-ci-summary` accepts Full Release Validation parent runs only.
491Diverged release-branch logs: `--first-parent` plus a bounded count.
492Stop watchers before ending the turn or switching strategy.
493
494Interpret state precisely:
495
496- `qualifying`: no decisive blocker yet; selected children are still active.
497- `blocked_diagnostics_running`: publication is blocked; Diagnostic Drain is
498 still collecting independent failures. Diagnose now, but do not retry until
499 the drain is terminal.
500- `passed`: all required policy and exact-child evidence passed.
501- `blocked_complete`: publication is blocked and all selected diagnostics are
502 terminal.
503- `orchestration_error`: GitHub API or collector failure prevented a verdict.
504 This is not a provenance mismatch. Recover the collector against the same
505 exact children; never redispatch tests to repair collection.
506- `cancelled_with_children`: the collector was cancelled while exact children
507 remained active.
508
509Read **advisory** entries separately from Release Decision. Windows/macOS
510cross-OS lanes retain their actual conclusions in the manifest and summary;
511`passed` does not mean those advisory lanes passed. Selected lanes still need
512terminal evidence, and filtered-out lanes are not run, never passed.
513
514The `full-release-diagnostics-<run-id>-<attempt>` artifact is the terminal
515failure and timing manifest. Use it after an early blocker instead of
516restarting `all` merely to discover what the still-running children found.
517The stable `full-release-execution-plan-<run-id>` artifact is the identity
518source within each collector attempt; retry attempts restore its immutable
519run-ID-cached bytes first.
520
521## Failure Triage
522
5231. Confirm parent SHA and child run IDs.
5242. List failed jobs only:
525 ```bash
526 gh run view <child-run-id> --repo openclaw/openclaw --json jobs \
527 --jq '.jobs[] | select(.conclusion=="failure" or .conclusion=="timed_out" or .conclusion=="cancelled") | [.databaseId,.name,.conclusion,.url] | @tsv'
528 ```
5293. Fetch one failed job log. If rate-limited, note reset time and avoid more REST calls.
5304. For secret-looking failures, validate a real completion from the same secret source before editing code. A successful model-list request is insufficient.
531 Claude CLI subscription credentials are a separate native auth path; prove
532 them in a clean-home CLI probe, never as a substitute for a required
533 Anthropic API-key lane.
5345. For live-cache failures, inspect whether it is missing/invalid key, empty text, provider refusal, timeout, or baseline miss. Do not weaken release gates without clear provider evidence.
5356. Classify before editing:
536 - confirmed product/code failure: fix the release branch, freeze a new Code
537 SHA, and invalidate product evidence
538 - harness/tooling/provenance failure: keep the Code SHA, fix the smallest
539 owning surface, and retry only the failed surface with the required Tooling
540 SHA
541 - infrastructure/credential failure: keep both SHAs, repair the external
542 prerequisite, and retry only the failed surface
543 - wrapper/monitor failure: keep the child and candidate identities; record
544 the wrapper result separately from the child result
545 - changelog/release-note failure: change only `CHANGELOG.md`, keep Code SHA
546 evidence, and repeat Release SHA proof
547 - publish child/registry selector failure: keep Release SHA and resume the
548 failed child; never rebuild an immutable version that already published
549 Only the first class changes the Code SHA. After one diagnosis/fix/narrow
550 retry, reassess instead of starting another all-group cycle.
5517. If a required PR CI run is capacity-stalled with queued jobs and no active
552 jobs, do not cancel unrelated work or accept a generic manual dispatch.
553 First verify the PR head carries the current fallback schema:
554 `gh api 'repos/openclaw/openclaw/contents/.github/workflows/ci.yml?ref=<pr-head-branch>'
555--jq .content | base64 --decode | rg -q 'pull_request_number:'`. If absent,
556 refresh the PR head from `main` and use the new head SHA; let normal CI run
557 before considering another fallback.
558 From the PR head branch, dispatch the explicit exact-SHA fallback:
559 `gh workflow run ci.yml --repo openclaw/openclaw --ref <pr-head-branch> -f
560target_ref=<full-pr-sha> -f pull_request_number=<pr-number> -f
561include_android=true -f release_gate=true`.
562 It runs on GitHub-hosted runners and is accepted only when its run title is
563 `CI release gate <full-pr-sha>`. Record the stalled Blacksmith run and the
564 fallback run in release evidence.
565 If `Blacksmith Build Artifacts Testbox` is the only remaining required gate
566 and remains queued without a runner, that completed exact fallback may cover
567 it because CI's `build-artifacts` job already builds, packages, and smoke
568 tests the artifacts. Do not use this coverage after the artifact workflow
569 starts or completes non-successfully.
570
571## Evidence
572
573Record:
574
575- release lifecycle ledger: Code SHA, Release SHA, and Tooling SHA for regular
576 releases; canonical branch, exact SHA, and immutable tag for extended-stable
577- evidence-reuse policy, coverage policy, and complete changed-path set
578- active full parent run URL, attempt, workflow SHA, and any superseded parent
579 with the exact replacement reason
580- selected child run IDs and conclusions: CI, Release Checks, Plugin Prerelease, NPM Telegram, Product Performance; record deferred confidence as not run
581- Windows/macOS cross-OS advisory lane classifications and actual conclusions
582- performance comparison result versus earlier releases when available
583- targeted local proof commands
584- provider-secret preflight result
585- frozen-target compatibility repairs or omitted inapplicable scenarios, with
586 their source PRs and invariant
587- known gaps or unrelated failures
588
589For lessons and recovery patterns, read `references/release-ci-notes.md`.