OpenClaw Release CI
Use this with $release-openclaw-maintainer and $openclaw-testing when a release candidate needs full validation, install/update proof, live provider checks, or CI recovery.
Guardrails
- No version bump, tag, npm publish, GitHub release, or release promotion without explicit operator approval.
- After compaction, resume, or new steering, rewrite the effective goal and
current phase from the latest explicit operator instruction. Do not merge old
scope back into the active release.
- Hold the release scope once a release branch or Code SHA exists. Validate and
ship that exact release; do not turn moving
main into a second work queue.
- Record every active validation run as the immutable tuple Validation SHA +
Tooling SHA + rerun group. Validation SHA maps to the Code SHA for product validation or
the Release SHA for changelog-only validation; it is not a third release
identity. A branch or temporary ref is context and transport.
- Freeze the candidate SHA/ref and Tooling SHA/ref once. Main lineage authorizes
the initial Tooling SHA selection; it does not authorize replacing that
tooling after
main advances.
- Apply a release firebreak after the Code SHA is frozen. Admit only confirmed
product defects, package/provenance defects in the bytes to publish, security
defects, or failures that make publication impossible. Queue other findings
for postpublish confidence or the next beta.
- Use trusted
main workflow revisions as immutable dispatch sources. Do not
adopt newer main code, repair unrelated main CI, wait for broad main health,
or expand a release fix because the workflow source lives on main.
- Once publication binds the Tooling SHA to an exact protected lightweight
release-publish/<12sha>-<provenance-run> tag, that live tag-to-SHA mapping
remains authoritative when main advances. The suffix records tag-creation
provenance; it is not the current parent run id.
- Touch
main only for an operator-requested change or the smallest critical
main-owned blocker that prevents this release and cannot be handled from the
release branch. If the required main landing policy is blocked by unrelated
main failures, report that blocker and keep independent release work moving
instead of healing broader main.
- Validate provider secrets before dispatching expensive full release matrices.
- Linux (
ubuntu) cross-OS lanes gate publication for beta, stable, and full.
Windows/macOS cross-OS lanes run in parallel as advisory coverage. Record
their actual pass/fail conclusions; failures do not block Release Decision,
npm publication, or pnpm release:candidate. Keep normal CI, npm
qualification, Docker, Package Acceptance, performance, and soak gates intact.
- macOS app signing/notarization/appcast and Windows Hub asset promotion run
in parallel with or after npm publication and never delay npm or GitHub
finalization. Their own qualification and artifact gates still apply; track
selected platforms through verified assets and updater evidence separately.
- Do not set GitHub secrets from unvalidated 1Password candidates. If a candidate returns 401/403, leave the existing secret alone and report the exact missing provider.
- Use
$one-password for secret reads/writes: one persistent tmux session, targeted items only, no secret output.
- Watch one parent run plus compact child summaries. Avoid broad
gh run view polling loops; REST quota is easy to burn.
- Fetch logs only for failed or currently-blocking jobs. If quota is low, stop polling and wait for reset.
- Treat live-provider flakes separately from code failures: prove key validity, provider HTTP status, retry evidence, and exact failing lane before editing code.
- A model-list response proves authentication, not billing or inference
entitlement. Mandatory live providers must pass a real completion probe
before release dispatch. Fix the credential first; do not add an alternate
auth path merely to bypass a failed release credential.
- Full Release Validation separates exact-child dispatch, Release Decision,
and Diagnostic Drain. With
fail_fast=false, it makes zero child
cancellation calls; Diagnostic Drain follows every selected child to
terminal unless the collector itself is cancelled or loses GitHub API
access. With
fail_fast=true, Release Decision may cancel only the exact still-active
child that owns a blocking failure.
- Same-parent continuation requires the original root to have been dispatched
with
fail_fast=false. The controller verifies that exact logged input
before any rerun mutation.
- A parent that produced its own sealed candidate artifacts cannot be continued:
GitHub reruns make those prior-attempt artifacts unavailable. Keep the
candidate and Tooling SHAs frozen, supersede that parent, and start a fresh
all-group Full Release Validation.
- After dispatch, one immutable execution-plan artifact records the original
parent attempt, exact child tuples and titles, selected coverage, gates, and
reuse identity. The same bytes are saved under an exact run-ID cache key.
Decision, Drain, manifest writing, evidence validation, and final verification
consume the artifact for their current attempt. A collector retry restores
the cached plan, validates it, re-uploads its artifact, and adopts the same
children; missing plan state is an orchestration failure, not permission to
reconstruct the plan or redispatch.
- Reused evidence is not trusted merely because plan sealing found it. Release
Decision repeats the sealed target SHA, evidence SHA, policy, changed paths,
selected run, root run, source manifest, trusted tooling identity, and
exact-child checks before returning
passed.
- Parent retries select the newest Decision and Drain artifacts independently;
both must bind the same immutable plan even when their source attempts differ.
- Child retries are part of the same immutable plan only when the child run ID,
workflow path, ref, Tooling SHA, dispatch title, event, target, candidate, and
validation inputs remain exact. Newer child attempts replace matching jobs;
jobs absent from a newer attempt carry forward. A duplicate job identity,
missing attempt, regressed attempt, or changed tuple fails closed.
- Use
pnpm frv status|continue --failed|verify for attempt-aware recovery.
The controller is stateless: the immutable execution plan, exact GitHub run
attempts, Diagnostic Drain, and final manifest are the only authorities. It
never writes a tag, package, registry entry, release candidate, or
publication.
- Post-merge controller proof must use the reviewed landed SHA on protected
main through the non-release FRV Proof Broker and FRV Proof Fixture.
Dispatch the broker with the merged pull request number and exact landed
commit. The broker must require that pull request's merge commit to match the
landed commit, prove the landed commit is identical to or an ancestor of its
trusted workflow SHA, and repeat authority, merge, and ancestry checks
immediately before rerunning the fixture.
Require the exact fixed no-op fixture run to advance from its intentional
attempt-one failure to an attempt-two pass. The broker must emit its receipt
without creating a release candidate, release artifact, publication,
repository ref, replacement parent, or other workflow mutation. This is the
hosted GitHub failed-job rerun proof; focused controller tests own immutable
plan eligibility, green-attempt preservation, same-parent collection, and
strict-verifier invocation. Never use a real Full Release Validation run for
this proof. See
Full Release Validation.
- Use one release operator, one transition-only watcher, and at most one
investigator for the current failed surface. Do not build audit-review-plan
trees around a single workflow transition.
- For regular beta/stable releases, Code SHA may already contain final notes
and serve as Release SHA. One successful fresh full parent may qualify both
roles and their exact publication bytes. If notes change afterward, a later
Release SHA may reuse product evidence only when its complete delta from
Code SHA is exactly
CHANGELOG.md; its changed bytes still need qualification.
- Extended-stable validates one exact branch tip; it does not reuse the regular
Code-SHA/Release-SHA evidence model.
- In a sparse worktree or Testbox source sync, first confirm
package.json,
pnpm-lock.yaml, and every source path the selected check reads. If any are
absent, that checkout cannot validate a release dependency or Docker lane:
stop and use the repo remote changed gate or a full task worktree. When the
inputs are present and a release fix changes package.json or
pnpm-lock.yaml, rebuild only the task-owned disposable box with
CI=true pnpm install --frozen-lockfile, then run an explicit
require.resolve() probe before Docker or focused tests. The CI flag permits
pnpm to recreate a prewarmed modules directory without an interactive
confirmation. Do not weaken the lockfile or label sparse-checkout failures
as product/Docker failures.
- If the candidate is rebased or its base SHA changes after warmup, stop the
task-owned box and warm a fresh one before testing. Testbox source sync is
relative to the warmed source tree; continuing can mix an old base file with
a new candidate diff and produce false lockfile or Docker failures.
- Reused Testboxes are provenance-gated after their first successful run.
Source-only edits may reuse the lease; base, dependency, wrapper, or Testbox
workflow drift requires a fresh lease. Do not set
OPENCLAW_TESTBOX_ALLOW_STALE=1 for release evidence.
- For a committed release candidate, warm the box with
blacksmith testbox warmup ... --ref <candidate-branch-or-sha>. Do not rely
on source sync to overlay committed branch changes onto the workflow's
default ref.
Run identity and retry budget
Record Validation SHA, Tooling SHA/ref, target context ref, parent run id,
attempt, and phase before watching or recovering Full Release Validation. Keep
Code SHA and Release SHA as lifecycle roles in the ledger; they may name the
same commit. Record the
immutable Release Publish parent receipt separately from tag provenance.
For the core and plugin npm mutations enforced by this foundation, re-read the
exact protected lightweight tag and revalidate the exact parent run tuple
immediately before each publish or dist-tag mutation. Reject a missing, moved,
annotated, or wrong-SHA tag; a repository, workflow, run id, attempt, tooling
identity, or parent-state mismatch; and any same-name branch. Never refresh
either identity from current main. Treat other privileged writers as blocked
until their dependent enforcement changes land.
Conceptual phases map to current inputs as follows:
beta-publish: release_profile=beta, run_release_soak=false
postpublish-confidence: published package inputs with
run_release_soak=true or explicit focused groups
stable-publish: release_profile=stable
An all run without soak for an actual beta package on its matching canonical
release branch or beta tag records coveragePolicy=npm-beta-v1. It keeps
Linux/macOS/Windows Node, Control UI, plugin, package, install/update,
Linux cross-OS, QA parity, runtime-pair/restart, and tool-coverage gates. Native app
CI, performance, and published-package Telegram are deferred to confidence.
Beta all without soak also defers Package Acceptance Telegram, including
beta-profile checks of main or alpha. Record deferred checks as not run,
never passed. Stable/full, soak, and focused groups retain their coverage;
selected children still require terminal evidence. An absent coverage policy
retains historical full behavior.
Keep at most one active parent for the same Validation SHA + Tooling SHA + rerun
group + release profile + effective soak coverage. Stable/full always include
soak. Distinct coverage profiles can run independently; concurrency does not
cancel an older exact child automatically.
Parent cancellation or timeout leaves adopted identity-checked children
running. The operator must cancel an exact child explicitly when it is no
longer useful. Do not infer a child identity from branch, title prefix, or
latest-run order.
Recover one failed surface with one diagnosis, one fix when needed, and one
narrow retry. Then reassess the release decision. Do not automatically
dispatch rerun_group=all.
For a supported parent, pnpm frv continue --failed --run <parent-run-id>
adopts any active newer child attempt, reruns failed child jobs in parallel,
leaves green children untouched, then reruns the parent once to restore the
immutable plan and seal a trusted all-group manifest. It does not start a
second child retry while an attempt is active. Each child or parent rerun
mutation is sent exactly once; ambiguous transport failures trigger only
bounded read reconciliation. The controller never repeats the mutation, and
provenance drift fails closed.
Inspect without mutation:
pnpm frv status --run <parent-run-id>
pnpm frv verify --run <successful-parent-run-id>
Parents whose immutable plan predates attempt-aware evidence cannot be
continued. Start a fresh all-group Full Release Validation; never reconstruct
old state or dispatch a replacement parent.
Controller retries are ci, plugin-prerelease, install-smoke,
cross-os, live-e2e, package, qa-parity, qa-live, npm-telegram,
or performance. Never use the removed release-checks handle. qa is
only a direct-child manual aggregate, not a controller retry API.
Filtered retries fail closed unless the filter belongs to the selected group.
All-group runs also accept cross_os_suite_filter: for example,
-f cross_os_suite_filter=ubuntu,macos excludes Windows. npm-stable-v1 and
npm-beta-v1 still qualify when advisory OS lanes are omitted, provided all
Linux suites remain selected and the other policy requirements hold.
Never turn an empty derived filter into an unfiltered broad run.
A new all-group parent is justified only when shared orchestration changed,
earlier evidence is invalid for the selected tuple, or the operator explicitly
requests it. Record the invalidating event.
Narrow child or group evidence does not by itself become publish
authorization. Keep it in the evidence ledger for the release owner to judge
against the current publish gate.
Preflight
Before full matrix dispatch, run both pnpm ui:i18n:check and
pnpm native:i18n:check against the frozen trusted target in approved isolation.
Bind both results to that exact SHA; either generated-locale drift blocks
dispatch. Keep target execution outside the trusted dispatch helper—do not
execute an arbitrary target checkout as helper code.
Before expensive full validation, also run pnpm ui:build on the same frozen
trusted target with its frozen dependencies in approved isolation, outside the
trusted dispatch helper. Record the target SHA with the successful production
build, precompressed-asset verification, and startup/largest-asset budget results;
any failure blocks fanout. Do not substitute a dev server or raise budgets to admit
the target.
For local full E2E proof, prepare the frozen, dependency-ready proof checkout
with private QA entries in the initial build:
OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build
Then run the selected E2E command with its normal readiness checks enabled.
scripts/lib/vitest-build-prerequisites.mts requests private QA entries;
scripts/run-node.mts triggers another full build when they are absent. This
preflight avoids rebuilding solely for missing_private_qa_dist. Keep the flag
scoped to this task-owned proof checkout and command. Publication package and
image bytes remain owned by the release workflows and their sealed artifacts.
Before full release validation:
node .agents/skills/release-openclaw-ci/scripts/verify-provider-secrets.mjs --required openai,anthropic,fireworks
gh api rate_limit --jq '.resources.core'
git status --short --branch
git rev-parse HEAD
1Password service-account values are the first source for release provider
preflight. Inject those exact targeted keys first, then run the verifier; use
ambient env only when it was already intentionally injected for this release.
The script prints only provider status and HTTP class, never tokens.
The Anthropic check performs a tiny message completion so exhausted or
non-billable credentials fail before the expensive release matrix.
Dispatch
An early standalone product-performance run is optional beta confidence. If
useful, start it against the frozen Code SHA in parallel with release work:
# Optional early beta confidence; stable/full use the required parent child.
fail_on_regression=false
gh workflow run openclaw-performance.yml \
--repo openclaw/openclaw \
--ref main \
-f target_ref=<code-sha> \
-f profile=release \
-f repeat=3 \
-f deep_profile=false \
-f live_openai_candidate=false \
-f fail_on_regression="$fail_on_regression"
- Do not add a separate mandatory prepublish wait for this optional beta signal.
- Compare available Kova, gateway startup, and CLI startup metrics with earlier
release evidence or clawgrit reports before publish/closeout.
- Call out any regression in the release proof. Treat a major regression as a
release blocker until it is fixed, waived by the operator, or proven to be
infrastructure noise.
- Full Release Validation requires blocking performance evidence for stable
and full profiles.
npm-beta-v1 defers the child; explicit performance
and soak-enabled beta runs retain advisory performance coverage. Every
selected performance child must finish and prove artifact-only publication.
Prefer an immutable trusted-main workflow revision, target the exact Code SHA:
- Keep trusted-workflow checks compatible with frozen release targets. If
main adds a target-owned guard script or package command after the release
branch cut, make the trusted workflow skip only when that target surface is
absent. Repair the smallest trusted-workflow compatibility issue only when it
blocks the release, then rerun validation. Do not port an unrelated runtime
refactor, heal other main failures, or mutate the release candidate just to
satisfy a newer main-only check.
TOOLING_SHA="<exact-main-ancestor-sha>"
node scripts/full-release-validation-at-sha.mjs \
--sha <code-sha> \
--target-ref release/YYYY.M.PATCH \
--workflow-sha "$TOOLING_SHA"
For regular release/* validation, never raw-dispatch the workflow without
target_context_ref (the helper's --target-ref records it). Canonical
release/* and extended-stable/* workflow refs remain supported routes, but
their Telegram child must retain the exact parent workflow ref and SHA through
OIDC and attestation. Trusted-workflow release-branch CI passes target_ref +
release_candidate_ref; never release_gate there — it requires workflow head
== target. (The PR-head ci.yml fallback below is a different dispatch and does
use release_gate=true.)
The release branch may advance after the Code SHA is frozen. The helper accepts
that frozen SHA only while it remains an ancestor of the canonical release
branch and its package version is either the branch's final version or a
matching beta prerelease. Alpha remains on the Tideclaw path with a matching
alpha branch and exact alpha tag. Extended-stable branches and all tags require
an exact package-version match.
Always pass the previously recorded full Tooling SHA for release-branch runs.
Never replace it with a fresh main lookup. The Tooling SHA must declare the
current release-isolation contract; older workflow revisions fail closed.
For immutable workflow proof on a moving main, use
pnpm ci:full-release --sha <code-sha> --target-ref release/YYYY.M.PATCH --workflow-sha <tooling-sha>. Its canonical release-ci/* ref keeps evidence reuse
enabled after proving the workflow commit is still on trusted main lineage.
Pass -f reuse_evidence=false only when the operator intentionally needs a
fresh full run.
If final notes were already committed before fresh full qualification, retain
that Code SHA as Release SHA and use the same successful parent/attempt and
its exact prepared bytes for candidate and publication checks. Required gates,
final channel-specific SDK review and acknowledgement still apply.
Only if notes change after qualification, commit exactly CHANGELOG.md and
optionally run the helper against the new Release SHA with reuse. That parent must report
policy=changelog-only-release-v1, evidenceSha=<code-sha>, and
changedPaths=["CHANGELOG.md"]; it should reuse the product matrix instead of
dispatching child lanes. Npm preflight and package/install acceptance still run
against the exact Release SHA and its new tarball bytes.
Current all-group FRV also owns read-only npm source/build/qualification and
Docker preparation. Use its successful run as preflight_run_id; the candidate
helper defaults to that run. Do not dispatch a second npm preflight unless
recovering historical separate evidence. Regular final qualification records
SDK reports for both beta and latest; review the acknowledgement for the
actual publication channel. Prepared descriptors live in publicationArtifacts in
the exact final manifest. Product evidence reuse never substitutes Code-SHA
package or image bytes for the final Release SHA. A parent that produced these
artifacts needs a fresh all-group FRV instead of same-parent continuation.
The SHA-pinned helper infers beta for matching beta release candidates and
exact alpha tags, and stable for stable/correction versions, then passes the
Validation SHA + Tooling SHA run identity. Canonical beta all without soak
uses npm-beta-v1; main, alpha, and non-beta targets do not qualify for that
policy. Run deferred native, performance, Telegram, broad live QA, and E2E as
postpublish confidence with the exact published package and
run_release_soak=true or explicit groups. Stable and full profiles force the
release soak. Native artifact publication still requires its own build,
signing, notarization, and promotion gates. Use a narrow rerun_group after
focused fixes; never widen automatically.
Publish with openclaw-release-publish.yml using release_profile=from-validation
unless a maintainer intentionally wants to cross-check a specific profile; the
publish workflow reads the effective profile from the full-validation manifest.
Extended-stable validation
Use one remote-only procedure for .33+ extended-stable validation. Keep these
four identities separate:
- Validation SHA: exact 40-character candidate commit to validate.
- Tooling SHA: exact trusted-main commit whose workflows and helpers run.
- Context ref: canonical
extended-stable/YYYY.M.33 branch containing the
candidate.
- Workflow transport ref: immutable
release-ci/<tooling-sha-prefix>-<unique-id> branch at the Tooling SHA.
GitHub workflow dispatch --ref accepts a branch or tag name, not a raw commit
SHA. Never raw-dispatch this validation or hand-assemble its identity inputs.
Use the checked helper exclusively:
VALIDATION_SHA="<exact-candidate-sha>"
TOOLING_SHA="<recorded-full-main-ancestor-sha>"
CONTEXT_REF="extended-stable/YYYY.M.33"
pnpm ci:full-release \
--sha "$VALIDATION_SHA" \
--target-ref "$CONTEXT_REF" \
--workflow-sha "$TOOLING_SHA" \
-f release_profile=stable \
-f run_release_soak=true \
-f fail_fast=false \
-f rerun_group=all \
-f reuse_evidence=false \
-f dispatch_release_evidence=false
The helper verifies both SHAs, creates the transport ref with the equivalent of
the following GitHub refs operation, and dispatches from that branch:
gh api --method POST repos/openclaw/openclaw/git/refs \
-f ref="refs/heads/release-ci/${TOOLING_SHA:0:12}-<unique-id>" \
-f sha="$TOOLING_SHA"
Do not run that operation separately. The helper also supplies
ref=$VALIDATION_SHA, expected_sha=$VALIDATION_SHA,
target_context_ref=$CONTEXT_REF, and this exact trusted identity:
{"fullRef":"refs/heads/main","ref":"main","sha":"<tooling-sha>"}
Outside this extended-stable procedure, a direct canonical-branch dispatch is
valid only when that branch's own head is both the Validation SHA and the
trusted workflow implementation to execute. It cannot use a different
trusted-main Tooling SHA. Current extended-stable validation requires distinct
trusted-main tooling, so it must use the immutable release-ci/* transport
above. Direct canonical-branch and mutable-main dispatches are not valid
alternatives for this procedure.
Accept only a complete rerun_group=all run with a supported exact-target
manifest. Bind its workflow SHA separately from the candidate SHA; require the
manifest target, package versions, saved run_attempt, and final tag to identify
the same candidate. Reject narrow runs, untrusted tooling, mismatched targets,
and earlier-attempt evidence.
Run the npm preflight separately from trusted main. Here tag is the exact
candidate SHA; it is an npm-preflight input, not the workflow transport ref:
gh workflow run openclaw-npm-release.yml \
--repo openclaw/openclaw \
--ref main \
-f tag="$VALIDATION_SHA" \
-f preflight_only=true \
-f npm_dist_tag=extended-stable \
-f release_candidate_branch="$CONTEXT_REF"
This standalone run is a supplemental validation-only preflight. Do not pass
its run ID as publication preflight_run_id: a main workflow head does not
have the canonical candidate branch/SHA identity required by that publication
input. Publication continues to use the Full Release Validation run's
manifest-bound integrated npm artifact and exact run attempt.
Product failures need an approved backport. Frozen-target tooling failures need
the smallest behavior-preserving repair. Provider, approval, runner, or log
races keep the candidate unchanged. Record repairs and superseded runs; any
branch change requires a new complete parent. Omit only an explicitly
unsupported frozen-target scenario, never a required behavior or package.
Watch
Use the transition-only summary watcher instead of repeated raw polling:
node scripts/release-ci-summary.mjs <full-release-run-id> --watch
Do not start this watcher when the SHA-pinned helper is still the foreground
owner. The helper reads the exact Release Decision artifact itself. On
blocked_diagnostics_running, it exits nonzero immediately, keeps the temporary
refs, and leaves Diagnostic Drain collecting the remaining terminal evidence.
The watcher behaves the same way for separately dispatched parents: it reports
the Release Decision blocker once and exits while the drain continues.
For a one-shot snapshot:
node scripts/release-ci-summary.mjs <full-release-run-id>
release-ci-summary accepts Full Release Validation parent runs only.
Diverged release-branch logs: --first-parent plus a bounded count.
Stop watchers before ending the turn or switching strategy.
Interpret state precisely:
qualifying: no decisive blocker yet; selected children are still active.
blocked_diagnostics_running: publication is blocked; Diagnostic Drain is
still collecting independent failures. Diagnose now, but do not retry until
the drain is terminal.
passed: all required policy and exact-child evidence passed.
blocked_complete: publication is blocked and all selected diagnostics are
terminal.
orchestration_error: GitHub API or collector failure prevented a verdict.
This is not a provenance mismatch. Recover the collector against the same
exact children; never redispatch tests to repair collection.
cancelled_with_children: the collector was cancelled while exact children
remained active.
Read advisory entries separately from Release Decision. Windows/macOS
cross-OS lanes retain their actual conclusions in the manifest and summary;
passed does not mean those advisory lanes passed. Selected lanes still need
terminal evidence, and filtered-out lanes are not run, never passed.
The full-release-diagnostics-<run-id>-<attempt> artifact is the terminal
failure and timing manifest. Use it after an early blocker instead of
restarting all merely to discover what the still-running children found.
The stable full-release-execution-plan-<run-id> artifact is the identity
source within each collector attempt; retry attempts restore its immutable
run-ID-cached bytes first.
Failure Triage
- Confirm parent SHA and child run IDs.
- List failed jobs only:
gh run view <child-run-id> --repo openclaw/openclaw --json jobs \
--jq '.jobs[] | select(.conclusion=="failure" or .conclusion=="timed_out" or .conclusion=="cancelled") | [.databaseId,.name,.conclusion,.url] | @tsv'
- Fetch one failed job log. If rate-limited, note reset time and avoid more REST calls.
- For secret-looking failures, validate a real completion from the same secret source before editing code. A successful model-list request is insufficient.
Claude CLI subscription credentials are a separate native auth path; prove
them in a clean-home CLI probe, never as a substitute for a required
Anthropic API-key lane.
- For live-cache failures, inspect whether it is missing/invalid key, empty text, provider refusal, timeout, or baseline miss. Do not weaken release gates without clear provider evidence.
- Classify before editing:
- confirmed product/code failure: fix the release branch, freeze a new Code
SHA, and invalidate product evidence
- harness/tooling/provenance failure: keep the Code SHA, fix the smallest
owning surface, and retry only the failed surface with the required Tooling
SHA
- infrastructure/credential failure: keep both SHAs, repair the external
prerequisite, and retry only the failed surface
- wrapper/monitor failure: keep the child and candidate identities; record
the wrapper result separately from the child result
- changelog/release-note failure: change only
CHANGELOG.md, keep Code SHA
evidence, and repeat Release SHA proof
- publish child/registry selector failure: keep Release SHA and resume the
failed child; never rebuild an immutable version that already published
Only the first class changes the Code SHA. After one diagnosis/fix/narrow
retry, reassess instead of starting another all-group cycle.
- If a required PR CI run is capacity-stalled with queued jobs and no active
jobs, do not cancel unrelated work or accept a generic manual dispatch.
First verify the PR head carries the current fallback schema:
gh api 'repos/openclaw/openclaw/contents/.github/workflows/ci.yml?ref=<pr-head-branch>' --jq .content | base64 --decode | rg -q 'pull_request_number:'. If absent,
refresh the PR head from main and use the new head SHA; let normal CI run
before considering another fallback.
From the PR head branch, dispatch the explicit exact-SHA fallback:
gh workflow run ci.yml --repo openclaw/openclaw --ref <pr-head-branch> -f target_ref=<full-pr-sha> -f pull_request_number=<pr-number> -f include_android=true -f release_gate=true.
It runs on GitHub-hosted runners and is accepted only when its run title is
CI release gate <full-pr-sha>. Record the stalled Blacksmith run and the
fallback run in release evidence.
If Blacksmith Build Artifacts Testbox is the only remaining required gate
and remains queued without a runner, that completed exact fallback may cover
it because CI's build-artifacts job already builds, packages, and smoke
tests the artifacts. Do not use this coverage after the artifact workflow
starts or completes non-successfully.
Evidence
Record:
- release lifecycle ledger: Code SHA, Release SHA, and Tooling SHA for regular
releases; canonical branch, exact SHA, and immutable tag for extended-stable
- evidence-reuse policy, coverage policy, and complete changed-path set
- active full parent run URL, attempt, workflow SHA, and any superseded parent
with the exact replacement reason
- selected child run IDs and conclusions: CI, Release Checks, Plugin Prerelease, NPM Telegram, Product Performance; record deferred confidence as not run
- Windows/macOS cross-OS advisory lane classifications and actual conclusions
- performance comparison result versus earlier releases when available
- targeted local proof commands
- provider-secret preflight result
- frozen-target compatibility repairs or omitted inapplicable scenarios, with
their source PRs and invariant
- known gaps or unrelated failures
For lessons and recovery patterns, read references/release-ci-notes.md.
1---
2name: release-openclaw-ci
3description: "Run, watch, debug, and summarize OpenClaw full release CI, release checks, live provider gates, install/update proofs, and release-secret preflights."
4metadata:
5 source: https://skillmd.com/skills/openclaw/release-openclaw-ci
6---
7
8# OpenClaw Release CI
9
10Use this with `$release-openclaw-maintainer` and `$openclaw-testing` when a release candidate needs full validation, install/update proof, live provider checks, or CI recovery.
11
12## Guardrails
13
14- No version bump, tag, npm publish, GitHub release, or release promotion without explicit operator approval.
15- After compaction, resume, or new steering, rewrite the effective goal and
16 current phase from the latest explicit operator instruction. Do not merge old
17 scope back into the active release.
18- Hold the release scope once a release branch or Code SHA exists. Validate and
19 ship that exact release; do not turn moving `main` into a second work queue.
20- Record every active validation run as the immutable tuple **Validation SHA +
21 Tooling SHA + rerun group**. Validation SHA maps to the Code SHA for product validation or
22 the Release SHA for changelog-only validation; it is not a third release
23 identity. A branch or temporary ref is context and transport.
24- Freeze the candidate SHA/ref and Tooling SHA/ref once. Main lineage authorizes
25 the initial Tooling SHA selection; it does not authorize replacing that
26 tooling after `main` advances.
27- Apply a release firebreak after the Code SHA is frozen. Admit only confirmed
28 product defects, package/provenance defects in the bytes to publish, security
29 defects, or failures that make publication impossible. Queue other findings
30 for postpublish confidence or the next beta.
31- Use trusted `main` workflow revisions as immutable dispatch sources. Do not
32 adopt newer main code, repair unrelated main CI, wait for broad main health,
33 or expand a release fix because the workflow source lives on `main`.
34- Once publication binds the Tooling SHA to an exact protected lightweight
35 `release-publish/<12sha>-<provenance-run>` tag, that live tag-to-SHA mapping
36 remains authoritative when `main` advances. The suffix records tag-creation
37 provenance; it is not the current parent run id.
38- Touch `main` only for an operator-requested change or the smallest critical
39 main-owned blocker that prevents this release and cannot be handled from the
40 release branch. If the required main landing policy is blocked by unrelated
41 main failures, report that blocker and keep independent release work moving
42 instead of healing broader main.
43- Validate provider secrets before dispatching expensive full release matrices.
44- Linux (`ubuntu`) cross-OS lanes gate publication for beta, stable, and full.
45 Windows/macOS cross-OS lanes run in parallel as advisory coverage. Record
46 their actual pass/fail conclusions; failures do not block Release Decision,
47 npm publication, or `pnpm release:candidate`. Keep normal CI, npm
48 qualification, Docker, Package Acceptance, performance, and soak gates intact.
49- macOS app signing/notarization/appcast and Windows Hub asset promotion run
50 in parallel with or after npm publication and never delay npm or GitHub
51 finalization. Their own qualification and artifact gates still apply; track
52 selected platforms through verified assets and updater evidence separately.
53- Do not set GitHub secrets from unvalidated 1Password candidates. If a candidate returns 401/403, leave the existing secret alone and report the exact missing provider.
54- Use `$one-password` for secret reads/writes: one persistent tmux session, targeted items only, no secret output.
55- Watch one parent run plus compact child summaries. Avoid broad `gh run view` polling loops; REST quota is easy to burn.
56- Fetch logs only for failed or currently-blocking jobs. If quota is low, stop polling and wait for reset.
57- Treat live-provider flakes separately from code failures: prove key validity, provider HTTP status, retry evidence, and exact failing lane before editing code.
58- A model-list response proves authentication, not billing or inference
59 entitlement. Mandatory live providers must pass a real completion probe
60 before release dispatch. Fix the credential first; do not add an alternate
61 auth path merely to bypass a failed release credential.
62- Full Release Validation separates exact-child dispatch, Release Decision,
63 and Diagnostic Drain. With `fail_fast=false`, it makes zero child
64 cancellation calls; Diagnostic Drain follows every selected child to
65 terminal unless the collector itself is cancelled or loses GitHub API
66 access. With
67 `fail_fast=true`, Release Decision may cancel only the exact still-active
68 child that owns a blocking failure.
69- Same-parent continuation requires the original root to have been dispatched
70 with `fail_fast=false`. The controller verifies that exact logged input
71 before any rerun mutation.
72- A parent that produced its own sealed candidate artifacts cannot be continued:
73 GitHub reruns make those prior-attempt artifacts unavailable. Keep the
74 candidate and Tooling SHAs frozen, supersede that parent, and start a fresh
75 all-group Full Release Validation.
76- After dispatch, one immutable execution-plan artifact records the original
77 parent attempt, exact child tuples and titles, selected coverage, gates, and
78 reuse identity. The same bytes are saved under an exact run-ID cache key.
79 Decision, Drain, manifest writing, evidence validation, and final verification
80 consume the artifact for their current attempt. A collector retry restores
81 the cached plan, validates it, re-uploads its artifact, and adopts the same
82 children; missing plan state is an orchestration failure, not permission to
83 reconstruct the plan or redispatch.
84- Reused evidence is not trusted merely because plan sealing found it. Release
85 Decision repeats the sealed target SHA, evidence SHA, policy, changed paths,
86 selected run, root run, source manifest, trusted tooling identity, and
87 exact-child checks before returning `passed`.
88- Parent retries select the newest Decision and Drain artifacts independently;
89 both must bind the same immutable plan even when their source attempts differ.
90- Child retries are part of the same immutable plan only when the child run ID,
91 workflow path, ref, Tooling SHA, dispatch title, event, target, candidate, and
92 validation inputs remain exact. Newer child attempts replace matching jobs;
93 jobs absent from a newer attempt carry forward. A duplicate job identity,
94 missing attempt, regressed attempt, or changed tuple fails closed.
95- Use `pnpm frv status|continue --failed|verify` for attempt-aware recovery.
96 The controller is stateless: the immutable execution plan, exact GitHub run
97 attempts, Diagnostic Drain, and final manifest are the only authorities. It
98 never writes a tag, package, registry entry, release candidate, or
99 publication.
100- Post-merge controller proof must use the reviewed landed SHA on protected
101 `main` through the non-release `FRV Proof Broker` and `FRV Proof Fixture`.
102 Dispatch the broker with the merged pull request number and exact landed
103 commit. The broker must require that pull request's merge commit to match the
104 landed commit, prove the landed commit is identical to or an ancestor of its
105 trusted workflow SHA, and repeat authority, merge, and ancestry checks
106 immediately before rerunning the fixture.
107 Require the exact fixed no-op fixture run to advance from its intentional
108 attempt-one failure to an attempt-two pass. The broker must emit its receipt
109 without creating a release candidate, release artifact, publication,
110 repository ref, replacement parent, or other workflow mutation. This is the
111 hosted GitHub failed-job rerun proof; focused controller tests own immutable
112 plan eligibility, green-attempt preservation, same-parent collection, and
113 strict-verifier invocation. Never use a real Full Release Validation run for
114 this proof. See
115 [Full Release Validation](/reference/full-release-validation#post-merge-continuation-proof).
116- Use one release operator, one transition-only watcher, and at most one
117 investigator for the current failed surface. Do not build audit-review-plan
118 trees around a single workflow transition.
119- For regular beta/stable releases, Code SHA may already contain final notes
120 and serve as Release SHA. One successful fresh full parent may qualify both
121 roles and their exact publication bytes. If notes change afterward, a later
122 Release SHA may reuse product evidence only when its complete delta from
123 Code SHA is exactly `CHANGELOG.md`; its changed bytes still need qualification.
124- Extended-stable validates one exact branch tip; it does not reuse the regular
125 Code-SHA/Release-SHA evidence model.
126- In a sparse worktree or Testbox source sync, first confirm `package.json`,
127 `pnpm-lock.yaml`, and every source path the selected check reads. If any are
128 absent, that checkout cannot validate a release dependency or Docker lane:
129 stop and use the repo remote changed gate or a full task worktree. When the
130 inputs are present and a release fix changes `package.json` or
131 `pnpm-lock.yaml`, rebuild only the task-owned disposable box with
132 `CI=true pnpm install --frozen-lockfile`, then run an explicit
133 `require.resolve()` probe before Docker or focused tests. The CI flag permits
134 pnpm to recreate a prewarmed modules directory without an interactive
135 confirmation. Do not weaken the lockfile or label sparse-checkout failures
136 as product/Docker failures.
137- If the candidate is rebased or its base SHA changes after warmup, stop the
138 task-owned box and warm a fresh one before testing. Testbox source sync is
139 relative to the warmed source tree; continuing can mix an old base file with
140 a new candidate diff and produce false lockfile or Docker failures.
141- Reused Testboxes are provenance-gated after their first successful run.
142 Source-only edits may reuse the lease; base, dependency, wrapper, or Testbox
143 workflow drift requires a fresh lease. Do not set
144 `OPENCLAW_TESTBOX_ALLOW_STALE=1` for release evidence.
145- For a committed release candidate, warm the box with
146 `blacksmith testbox warmup ... --ref <candidate-branch-or-sha>`. Do not rely
147 on source sync to overlay committed branch changes onto the workflow's
148 default ref.
149
150## Run identity and retry budget
151
152Record Validation SHA, Tooling SHA/ref, target context ref, parent run id,
153attempt, and phase before watching or recovering Full Release Validation. Keep
154Code SHA and Release SHA as lifecycle roles in the ledger; they may name the
155same commit. Record the
156immutable Release Publish parent receipt separately from tag provenance.
157
158For the core and plugin npm mutations enforced by this foundation, re-read the
159exact protected lightweight tag and revalidate the exact parent run tuple
160immediately before each publish or dist-tag mutation. Reject a missing, moved,
161annotated, or wrong-SHA tag; a repository, workflow, run id, attempt, tooling
162identity, or parent-state mismatch; and any same-name branch. Never refresh
163either identity from current `main`. Treat other privileged writers as blocked
164until their dependent enforcement changes land.
165
166- Conceptual phases map to current inputs as follows:
167 - `beta-publish`: `release_profile=beta`, `run_release_soak=false`
168 - `postpublish-confidence`: published package inputs with
169 `run_release_soak=true` or explicit focused groups
170 - `stable-publish`: `release_profile=stable`
171- An `all` run without soak for an actual beta package on its matching canonical
172 release branch or beta tag records `coveragePolicy=npm-beta-v1`. It keeps
173 Linux/macOS/Windows Node, Control UI, plugin, package, install/update,
174 Linux cross-OS, QA parity, runtime-pair/restart, and tool-coverage gates. Native app
175 CI, performance, and published-package Telegram are deferred to confidence.
176 Beta `all` without soak also defers Package Acceptance Telegram, including
177 beta-profile checks of `main` or alpha. Record deferred checks as not run,
178 never passed. Stable/full, soak, and focused groups retain their coverage;
179 selected children still require terminal evidence. An absent coverage policy
180 retains historical full behavior.
181- Keep at most one active parent for the same Validation SHA + Tooling SHA + rerun
182 group + release profile + effective soak coverage. Stable/full always include
183 soak. Distinct coverage profiles can run independently; concurrency does not
184 cancel an older exact child automatically.
185- Parent cancellation or timeout leaves adopted identity-checked children
186 running. The operator must cancel an exact child explicitly when it is no
187 longer useful. Do not infer a child identity from branch, title prefix, or
188 latest-run order.
189- Recover one failed surface with one diagnosis, one fix when needed, and one
190 narrow retry. Then reassess the release decision. Do not automatically
191 dispatch `rerun_group=all`.
192- For a supported parent, `pnpm frv continue --failed --run <parent-run-id>`
193 adopts any active newer child attempt, reruns failed child jobs in parallel,
194 leaves green children untouched, then reruns the parent once to restore the
195 immutable plan and seal a trusted all-group manifest. It does not start a
196 second child retry while an attempt is active. Each child or parent rerun
197 mutation is sent exactly once; ambiguous transport failures trigger only
198 bounded read reconciliation. The controller never repeats the mutation, and
199 provenance drift fails closed.
200- Inspect without mutation:
201
202 ```bash
203 pnpm frv status --run <parent-run-id>
204 pnpm frv verify --run <successful-parent-run-id>
205 ```
206
207- Parents whose immutable plan predates attempt-aware evidence cannot be
208 continued. Start a fresh all-group Full Release Validation; never reconstruct
209 old state or dispatch a replacement parent.
210
211- Controller retries are `ci`, `plugin-prerelease`, `install-smoke`,
212 `cross-os`, `live-e2e`, `package`, `qa-parity`, `qa-live`, `npm-telegram`,
213 or `performance`. Never use the removed `release-checks` handle. `qa` is
214 only a direct-child manual aggregate, not a controller retry API.
215- Filtered retries fail closed unless the filter belongs to the selected group.
216 All-group runs also accept `cross_os_suite_filter`: for example,
217 `-f cross_os_suite_filter=ubuntu,macos` excludes Windows. `npm-stable-v1` and
218 `npm-beta-v1` still qualify when advisory OS lanes are omitted, provided all
219 Linux suites remain selected and the other policy requirements hold.
220 Never turn an empty derived filter into an unfiltered broad run.
221- A new all-group parent is justified only when shared orchestration changed,
222 earlier evidence is invalid for the selected tuple, or the operator explicitly
223 requests it. Record the invalidating event.
224- Narrow child or group evidence does not by itself become publish
225 authorization. Keep it in the evidence ledger for the release owner to judge
226 against the current publish gate.
227
228## Preflight
229
230Before full matrix dispatch, run both `pnpm ui:i18n:check` and
231`pnpm native:i18n:check` against the frozen trusted target in approved isolation.
232Bind both results to that exact SHA; either generated-locale drift blocks
233dispatch. Keep target execution outside the trusted dispatch helper—do not
234execute an arbitrary target checkout as helper code.
235
236Before expensive full validation, also run `pnpm ui:build` on the same frozen
237trusted target with its frozen dependencies in approved isolation, outside the
238trusted dispatch helper. Record the target SHA with the successful production
239build, precompressed-asset verification, and startup/largest-asset budget results;
240any failure blocks fanout. Do not substitute a dev server or raise budgets to admit
241the target.
242
243For local full E2E proof, prepare the frozen, dependency-ready proof checkout
244with private QA entries in the initial build:
245
246```bash
247OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build
248```
249
250Then run the selected E2E command with its normal readiness checks enabled.
251`scripts/lib/vitest-build-prerequisites.mts` requests private QA entries;
252`scripts/run-node.mts` triggers another full build when they are absent. This
253preflight avoids rebuilding solely for `missing_private_qa_dist`. Keep the flag
254scoped to this task-owned proof checkout and command. Publication package and
255image bytes remain owned by the release workflows and their sealed artifacts.
256
257Before full release validation:
258
259```bash
260node .agents/skills/release-openclaw-ci/scripts/verify-provider-secrets.mjs --required openai,anthropic,fireworks
261gh api rate_limit --jq '.resources.core'
262git status --short --branch
263git rev-parse HEAD
264```
265
2661Password service-account values are the first source for release provider
267preflight. Inject those exact targeted keys first, then run the verifier; use
268ambient env only when it was already intentionally injected for this release.
269The script prints only provider status and HTTP class, never tokens.
270The Anthropic check performs a tiny message completion so exhausted or
271non-billable credentials fail before the expensive release matrix.
272
273## Dispatch
274
275An early standalone product-performance run is optional beta confidence. If
276useful, start it against the frozen Code SHA in parallel with release work:
277
278```bash
279# Optional early beta confidence; stable/full use the required parent child.
280fail_on_regression=false
281gh workflow run openclaw-performance.yml \
282 --repo openclaw/openclaw \
283 --ref main \
284 -f target_ref=<code-sha> \
285 -f profile=release \
286 -f repeat=3 \
287 -f deep_profile=false \
288 -f live_openai_candidate=false \
289 -f fail_on_regression="$fail_on_regression"
290```
291
292- Do not add a separate mandatory prepublish wait for this optional beta signal.
293- Compare available Kova, gateway startup, and CLI startup metrics with earlier
294 release evidence or clawgrit reports before publish/closeout.
295- Call out any regression in the release proof. Treat a major regression as a
296 release blocker until it is fixed, waived by the operator, or proven to be
297 infrastructure noise.
298- Full Release Validation requires blocking performance evidence for stable
299 and full profiles. `npm-beta-v1` defers the child; explicit `performance`
300 and soak-enabled beta runs retain advisory performance coverage. Every
301 selected performance child must finish and prove artifact-only publication.
302
303Prefer an immutable trusted-main workflow revision, target the exact Code SHA:
304
305- Keep trusted-workflow checks compatible with frozen release targets. If
306 `main` adds a target-owned guard script or package command after the release
307 branch cut, make the trusted workflow skip only when that target surface is
308 absent. Repair the smallest trusted-workflow compatibility issue only when it
309 blocks the release, then rerun validation. Do not port an unrelated runtime
310 refactor, heal other main failures, or mutate the release candidate just to
311 satisfy a newer `main`-only check.
312
313```bash
314TOOLING_SHA="<exact-main-ancestor-sha>"
315node scripts/full-release-validation-at-sha.mjs \
316 --sha <code-sha> \
317 --target-ref release/YYYY.M.PATCH \
318 --workflow-sha "$TOOLING_SHA"
319```
320
321For regular `release/*` validation, never raw-dispatch the workflow without
322`target_context_ref` (the helper's `--target-ref` records it). Canonical
323`release/*` and `extended-stable/*` workflow refs remain supported routes, but
324their Telegram child must retain the exact parent workflow ref and SHA through
325OIDC and attestation. Trusted-workflow release-branch CI passes `target_ref` +
326`release_candidate_ref`; never `release_gate` there — it requires workflow head
327== target. (The PR-head ci.yml fallback below is a different dispatch and does
328use `release_gate=true`.)
329
330The release branch may advance after the Code SHA is frozen. The helper accepts
331that frozen SHA only while it remains an ancestor of the canonical release
332branch and its package version is either the branch's final version or a
333matching beta prerelease. Alpha remains on the Tideclaw path with a matching
334alpha branch and exact alpha tag. Extended-stable branches and all tags require
335an exact package-version match.
336Always pass the previously recorded full Tooling SHA for release-branch runs.
337Never replace it with a fresh `main` lookup. The Tooling SHA must declare the
338current release-isolation contract; older workflow revisions fail closed.
339
340For immutable workflow proof on a moving `main`, use
341`pnpm ci:full-release --sha <code-sha> --target-ref
342release/YYYY.M.PATCH --workflow-sha <tooling-sha>`. Its canonical `release-ci/*` ref keeps evidence reuse
343enabled after proving the workflow commit is still on trusted `main` lineage.
344Pass `-f reuse_evidence=false` only when the operator intentionally needs a
345fresh full run.
346
347If final notes were already committed before fresh full qualification, retain
348that Code SHA as Release SHA and use the same successful parent/attempt and
349its exact prepared bytes for candidate and publication checks. Required gates,
350final channel-specific SDK review and acknowledgement still apply.
351
352Only if notes change after qualification, commit exactly `CHANGELOG.md` and
353optionally run the helper against the new Release SHA with reuse. That parent must report
354`policy=changelog-only-release-v1`, `evidenceSha=<code-sha>`, and
355`changedPaths=["CHANGELOG.md"]`; it should reuse the product matrix instead of
356dispatching child lanes. Npm preflight and package/install acceptance still run
357against the exact Release SHA and its new tarball bytes.
358
359Current all-group FRV also owns read-only npm source/build/qualification and
360Docker preparation. Use its successful run as `preflight_run_id`; the candidate
361helper defaults to that run. Do not dispatch a second npm preflight unless
362recovering historical separate evidence. Regular final qualification records
363SDK reports for both `beta` and `latest`; review the acknowledgement for the
364actual publication channel. Prepared descriptors live in `publicationArtifacts` in
365the exact final manifest. Product evidence reuse never substitutes Code-SHA
366package or image bytes for the final Release SHA. A parent that produced these
367artifacts needs a fresh all-group FRV instead of same-parent continuation.
368
369The SHA-pinned helper infers `beta` for matching beta release candidates and
370exact alpha tags, and `stable` for stable/correction versions, then passes the
371Validation SHA + Tooling SHA run identity. Canonical beta `all` without soak
372uses `npm-beta-v1`; `main`, alpha, and non-beta targets do not qualify for that
373policy. Run deferred native, performance, Telegram, broad live QA, and E2E as
374postpublish confidence with the exact published package and
375`run_release_soak=true` or explicit groups. Stable and full profiles force the
376release soak. Native artifact publication still requires its own build,
377signing, notarization, and promotion gates. Use a narrow `rerun_group` after
378focused fixes; never widen automatically.
379Publish with `openclaw-release-publish.yml` using `release_profile=from-validation`
380unless a maintainer intentionally wants to cross-check a specific profile; the
381publish workflow reads the effective profile from the full-validation manifest.
382
383### Extended-stable validation
384
385Use one remote-only procedure for `.33+` extended-stable validation. Keep these
386four identities separate:
387
388- **Validation SHA:** exact 40-character candidate commit to validate.
389- **Tooling SHA:** exact trusted-main commit whose workflows and helpers run.
390- **Context ref:** canonical `extended-stable/YYYY.M.33` branch containing the
391 candidate.
392- **Workflow transport ref:** immutable
393 `release-ci/<tooling-sha-prefix>-<unique-id>` branch at the Tooling SHA.
394
395GitHub workflow dispatch `--ref` accepts a branch or tag name, not a raw commit
396SHA. Never raw-dispatch this validation or hand-assemble its identity inputs.
397Use the checked helper exclusively:
398
399```bash
400VALIDATION_SHA="<exact-candidate-sha>"
401TOOLING_SHA="<recorded-full-main-ancestor-sha>"
402CONTEXT_REF="extended-stable/YYYY.M.33"
403pnpm ci:full-release \
404 --sha "$VALIDATION_SHA" \
405 --target-ref "$CONTEXT_REF" \
406 --workflow-sha "$TOOLING_SHA" \
407 -f release_profile=stable \
408 -f run_release_soak=true \
409 -f fail_fast=false \
410 -f rerun_group=all \
411 -f reuse_evidence=false \
412 -f dispatch_release_evidence=false
413```
414
415The helper verifies both SHAs, creates the transport ref with the equivalent of
416the following GitHub refs operation, and dispatches from that branch:
417
418```bash
419gh api --method POST repos/openclaw/openclaw/git/refs \
420 -f ref="refs/heads/release-ci/${TOOLING_SHA:0:12}-<unique-id>" \
421 -f sha="$TOOLING_SHA"
422```
423
424Do not run that operation separately. The helper also supplies
425`ref=$VALIDATION_SHA`, `expected_sha=$VALIDATION_SHA`,
426`target_context_ref=$CONTEXT_REF`, and this exact trusted identity:
427
428```text
429{"fullRef":"refs/heads/main","ref":"main","sha":"<tooling-sha>"}
430```
431
432Outside this extended-stable procedure, a direct canonical-branch dispatch is
433valid only when that branch's own head is both the Validation SHA and the
434trusted workflow implementation to execute. It cannot use a different
435trusted-main Tooling SHA. Current extended-stable validation requires distinct
436trusted-main tooling, so it must use the immutable `release-ci/*` transport
437above. Direct canonical-branch and mutable-`main` dispatches are not valid
438alternatives for this procedure.
439
440Accept only a complete `rerun_group=all` run with a supported exact-target
441manifest. Bind its workflow SHA separately from the candidate SHA; require the
442manifest target, package versions, saved `run_attempt`, and final tag to identify
443the same candidate. Reject narrow runs, untrusted tooling, mismatched targets,
444and earlier-attempt evidence.
445
446Run the npm preflight separately from trusted `main`. Here `tag` is the exact
447candidate SHA; it is an npm-preflight input, not the workflow transport ref:
448
449```bash
450gh workflow run openclaw-npm-release.yml \
451 --repo openclaw/openclaw \
452 --ref main \
453 -f tag="$VALIDATION_SHA" \
454 -f preflight_only=true \
455 -f npm_dist_tag=extended-stable \
456 -f release_candidate_branch="$CONTEXT_REF"
457```
458
459This standalone run is a supplemental validation-only preflight. Do not pass
460its run ID as publication `preflight_run_id`: a `main` workflow head does not
461have the canonical candidate branch/SHA identity required by that publication
462input. Publication continues to use the Full Release Validation run's
463manifest-bound integrated npm artifact and exact run attempt.
464
465Product failures need an approved backport. Frozen-target tooling failures need
466the smallest behavior-preserving repair. Provider, approval, runner, or log
467races keep the candidate unchanged. Record repairs and superseded runs; any
468branch change requires a new complete parent. Omit only an explicitly
469unsupported frozen-target scenario, never a required behavior or package.
470
471## Watch
472
473Use the transition-only summary watcher instead of repeated raw polling:
474
475```bash
476node scripts/release-ci-summary.mjs <full-release-run-id> --watch
477```
478
479Do not start this watcher when the SHA-pinned helper is still the foreground
480owner. The helper reads the exact Release Decision artifact itself. On
481`blocked_diagnostics_running`, it exits nonzero immediately, keeps the temporary
482refs, and leaves Diagnostic Drain collecting the remaining terminal evidence.
483The watcher behaves the same way for separately dispatched parents: it reports
484the Release Decision blocker once and exits while the drain continues.
485
486For a one-shot snapshot:
487
488```bash
489node scripts/release-ci-summary.mjs <full-release-run-id>
490```
491
492`release-ci-summary` accepts Full Release Validation parent runs only.
493Diverged release-branch logs: `--first-parent` plus a bounded count.
494Stop watchers before ending the turn or switching strategy.
495
496Interpret state precisely:
497
498- `qualifying`: no decisive blocker yet; selected children are still active.
499- `blocked_diagnostics_running`: publication is blocked; Diagnostic Drain is
500 still collecting independent failures. Diagnose now, but do not retry until
501 the drain is terminal.
502- `passed`: all required policy and exact-child evidence passed.
503- `blocked_complete`: publication is blocked and all selected diagnostics are
504 terminal.
505- `orchestration_error`: GitHub API or collector failure prevented a verdict.
506 This is not a provenance mismatch. Recover the collector against the same
507 exact children; never redispatch tests to repair collection.
508- `cancelled_with_children`: the collector was cancelled while exact children
509 remained active.
510
511Read **advisory** entries separately from Release Decision. Windows/macOS
512cross-OS lanes retain their actual conclusions in the manifest and summary;
513`passed` does not mean those advisory lanes passed. Selected lanes still need
514terminal evidence, and filtered-out lanes are not run, never passed.
515
516The `full-release-diagnostics-<run-id>-<attempt>` artifact is the terminal
517failure and timing manifest. Use it after an early blocker instead of
518restarting `all` merely to discover what the still-running children found.
519The stable `full-release-execution-plan-<run-id>` artifact is the identity
520source within each collector attempt; retry attempts restore its immutable
521run-ID-cached bytes first.
522
523## Failure Triage
524
5251. Confirm parent SHA and child run IDs.
5262. List failed jobs only:
527 ```bash
528 gh run view <child-run-id> --repo openclaw/openclaw --json jobs \
529 --jq '.jobs[] | select(.conclusion=="failure" or .conclusion=="timed_out" or .conclusion=="cancelled") | [.databaseId,.name,.conclusion,.url] | @tsv'
530 ```
5313. Fetch one failed job log. If rate-limited, note reset time and avoid more REST calls.
5324. For secret-looking failures, validate a real completion from the same secret source before editing code. A successful model-list request is insufficient.
533 Claude CLI subscription credentials are a separate native auth path; prove
534 them in a clean-home CLI probe, never as a substitute for a required
535 Anthropic API-key lane.
5365. For live-cache failures, inspect whether it is missing/invalid key, empty text, provider refusal, timeout, or baseline miss. Do not weaken release gates without clear provider evidence.
5376. Classify before editing:
538 - confirmed product/code failure: fix the release branch, freeze a new Code
539 SHA, and invalidate product evidence
540 - harness/tooling/provenance failure: keep the Code SHA, fix the smallest
541 owning surface, and retry only the failed surface with the required Tooling
542 SHA
543 - infrastructure/credential failure: keep both SHAs, repair the external
544 prerequisite, and retry only the failed surface
545 - wrapper/monitor failure: keep the child and candidate identities; record
546 the wrapper result separately from the child result
547 - changelog/release-note failure: change only `CHANGELOG.md`, keep Code SHA
548 evidence, and repeat Release SHA proof
549 - publish child/registry selector failure: keep Release SHA and resume the
550 failed child; never rebuild an immutable version that already published
551 Only the first class changes the Code SHA. After one diagnosis/fix/narrow
552 retry, reassess instead of starting another all-group cycle.
5537. If a required PR CI run is capacity-stalled with queued jobs and no active
554 jobs, do not cancel unrelated work or accept a generic manual dispatch.
555 First verify the PR head carries the current fallback schema:
556 `gh api 'repos/openclaw/openclaw/contents/.github/workflows/ci.yml?ref=<pr-head-branch>'
557--jq .content | base64 --decode | rg -q 'pull_request_number:'`. If absent,
558 refresh the PR head from `main` and use the new head SHA; let normal CI run
559 before considering another fallback.
560 From the PR head branch, dispatch the explicit exact-SHA fallback:
561 `gh workflow run ci.yml --repo openclaw/openclaw --ref <pr-head-branch> -f
562target_ref=<full-pr-sha> -f pull_request_number=<pr-number> -f
563include_android=true -f release_gate=true`.
564 It runs on GitHub-hosted runners and is accepted only when its run title is
565 `CI release gate <full-pr-sha>`. Record the stalled Blacksmith run and the
566 fallback run in release evidence.
567 If `Blacksmith Build Artifacts Testbox` is the only remaining required gate
568 and remains queued without a runner, that completed exact fallback may cover
569 it because CI's `build-artifacts` job already builds, packages, and smoke
570 tests the artifacts. Do not use this coverage after the artifact workflow
571 starts or completes non-successfully.
572
573## Evidence
574
575Record:
576
577- release lifecycle ledger: Code SHA, Release SHA, and Tooling SHA for regular
578 releases; canonical branch, exact SHA, and immutable tag for extended-stable
579- evidence-reuse policy, coverage policy, and complete changed-path set
580- active full parent run URL, attempt, workflow SHA, and any superseded parent
581 with the exact replacement reason
582- selected child run IDs and conclusions: CI, Release Checks, Plugin Prerelease, NPM Telegram, Product Performance; record deferred confidence as not run
583- Windows/macOS cross-OS advisory lane classifications and actual conclusions
584- performance comparison result versus earlier releases when available
585- targeted local proof commands
586- provider-secret preflight result
587- frozen-target compatibility repairs or omitted inapplicable scenarios, with
588 their source PRs and invariant
589- known gaps or unrelated failures
590
591For lessons and recovery patterns, read `references/release-ci-notes.md`.