Happier CI Stabilize

Happier CI Stabilization

Drive a failing CI or release run to an evidence-backed terminal result with the fewest expensive reruns. This skill owns failure collection, failure clustering, stabilization sequencing, CI cleanup decisions, monitoring, and recovery selection. It does not grant release authority or replace the test-quality rules in .agents/skills/happier-testing.

Use the existing owners

• Read and apply .agents/skills/happier-testing/SKILL.md before changing tests, fixtures, mocks, testkits, timeouts, or lane selection.
• Use .agents/skills/happier-implement/SKILL.md for repository corrections and its RED -> GREEN requirement for behavior changes.
• Resolve privileged publication through .agents/skills/happier-release/SKILL.md; this skill never invents release authority.
• After a coherent validated 0.2 correction, use .agents/skills/happier-port-0-2-to-0-3/SKILL.md once for an evidence-backed 0.3 disposition.

Read failure-collection.md for a failing run, ci-cleanup.md before simplifying CI/tests, and nightly-recovery.md before rerunning or resuming a release.

One stabilization loop

1. Bind the attempt. Record repository, workflow, run ID, attempt, event, head SHA, branch, and status. Never diagnose “latest” after a branch moves.
2. Collect once. Let independent safe jobs reach terminal state. Then run scripts/collect-actions-failures.mjs; retain complete failed-job logs under /tmp and use its compact summary. Stop early only for an unsafe mutation, conflicting publisher, or proven wedge blocking a corrected run.
3. Cluster causes. Collapse aggregators and repeated symptoms into the earliest causal signature and canonical owner. Classify each cluster as product, test, harness/mock, release control/configuration, external contract, infrastructure/resource, or inconclusive. A timeout is only a symptom.
4. Correct one batch. Reproduce every deterministic cluster that can be exercised locally. For each behavior change, prove the smallest meaningful RED, fix the canonical owner, and remove or update only coverage invalidated by that cause. Run all focused fixes together, then each affected lane.
5. Run canonical CI once. Use the exact coherent SHA. The manual Blacksmith runner-pool input may accelerate approved non-secret Linux lanes; it selects a backend for the same reusable workflow and must not create a copied CI graph.
6. Recover the cheapest safe way. Choose a native failed-job rerun, immutable-candidate resume, or fresh release using nightly-recovery.md. Preserve successful sibling candidates when their source and packaging identity remain valid.
7. Use one foreground monitor. Keep exactly one poller bound to the run and attempt. Poll long setup, suites, builds, signing, notarization, store submission, and publication every 5-20 minutes. Long duration alone is not failure evidence.
8. Close from terminal evidence. Require canonical CI success for the exact SHA. For a nightly, also inspect the terminal status artifact, immutable identities, required validation, promoted-reference verification, and requested side lanes. A green top-level badge alone is insufficient.

Iterate narrowly; certify once

Do not enqueue the entire graph after every correction. After one complete failure collection:

1. reproduce each deterministic cluster with the smallest owner-level local command;
2. run the affected package lane locally once the focused fixes are green;
3. when the local machine cannot represent the hosted boundary, dispatch tests-dispatch.yml with profile=custom, only the affected custom_checks, and exact ui_e2e_specs when applicable;
4. select a Blacksmith Linux pool only for eligible non-secret Linux work and keep macOS, Windows, self-hosted, credentialed, and release-mutation lanes on their canonical runners;
5. after the correction batch is coherent, run the required full/release profile once on the exact final SHA.

A GitHub native failed-job rerun is for the same SHA after a safe transient failure. It cannot validate code that exists only in a newer SHA. A custom dispatch on a corrected SHA is fast diagnostic evidence; it does not replace the final exact-SHA profile required by release policy.

Use this exact 0.2 manual-dispatch shape for a focused hosted diagnostic, replacing only the check list and optional UI E2E spec list with values accepted by the current workflow:

gh workflow run tests-dispatch.yml \
  --repo happier-dev/happier \
  --ref dev \
  -f profile=custom \
  -f runner_pool=github \
  -f custom_checks=release_contracts \
  -f installers_channel=stable \
  -f providers_preset=all \
  -f providers_tier=smoke

--ref selects the workflow ref, not an arbitrary checkout SHA. Immediately bind the created run ID and prove its headSha equals the intended 40-character commit before using the run as exact-SHA evidence:

gh run view <run-id> --repo happier-dev/happier \
  --json databaseId,attempt,event,headBranch,headSha,status,conclusion,workflowName,url

For a same-workflow-SHA safe transient, retain successful jobs and rerun only failures and their dependents:

gh run rerun <run-id> --repo happier-dev/happier --failed

Blacksmith never fails over automatically. If an explicitly approved Blacksmith run is unavailable or exhausted, start a new manual dispatch with the same profile/check/spec inputs and runner_pool=github; do not claim that the original run migrated pools. Blacksmith is never the pool for macOS, Windows, self-hosted, secret-bearing, or release-mutation jobs.

Keep source CI and release admission separate

• Source CI proves code correctness once for an exact SHA and emits or identifies explicit exact-SHA evidence.
• Release preflight cheaply proves that the already-tested SHA can be released with the requested channel, versions, notes, protocol, tools, credentials, and platform configuration.
• A nightly consumes the explicit successful ci_run_id/attestation after verifying its repository, workflow, event, branch, status, and exact SHA. It does not rerun full source CI.
• If explicit evidence is unavailable, use only the repository's canonical unattended lookup; do not create another scanner or occupy a release runner watching CI.

Expose every reachable failure

• Independent test jobs should not depend on unrelated test jobs.
• Independent checks inside one job may use continue-on-error only when a final if: always() aggregator fails the job if any required check failed.
• Diagnostic artifact upload and terminal status projection should use if: always() where safe.
• Publication, promotion, signing, destructive mutation, and security/trust gates remain fail-closed and must not continue merely to collect more errors.
• A collector exposes all reachable failures. A downstream job gated on a missing candidate cannot be meaningfully tested until that prerequisite exists; do not label this unavoidable dependency as hidden CI failure.

Move cheap deterministic contracts, workflow parsing, generated-output checks, and configuration preflights before expensive work. Preflight calls the canonical owner or inspects its output; it does not duplicate source CI or production decisions.

Stop conditions

Stop and report rather than retry when:

• the origin run is still active and resume validation requires a terminal status artifact;
• source or candidate bytes changed but the proposed recovery would reuse old artifacts;
• a release mutation returned an ambiguous failure and current external state has not been reconciled;
• expected behavior is a product decision rather than an observable current contract;
• logs are unavailable and the remaining evidence cannot distinguish product, test, harness, or infrastructure failure.

Handoff

Report:

• exact run/attempt/SHA and whether collection was complete;
• every root-cause cluster and its classification;
• focused RED/GREEN evidence and broader lanes actually run;
• tests, mocks, timeouts, or workflow gates consolidated/removed and why coverage did not weaken;
• recovery mechanism used and what work it preserved;
• terminal CI/release evidence and residual risk.