CI Monitor

CI Monitor Lifecycle

• The CI step produces findings, not fixes: a settled observation returns one finding per issue with an action (ciObservationFindings, internal/pipeline/steps/ci_findings.go) and the executor's shared auto-fix loop drives fix rounds (repairFromFindings in ci_fix.go consumes sctx.PreviousFindings), so auto_fix.ci is counted by the executor from round history and a review-bot check (scm.ReviewBots, identified by scm.Check.App) parks as ask-user comment findings without spending a round. A published repair keeps monitoring inside the fix round and must call sctx.MarkRunning() so status returns to running for the readiness consumers. AGENTS.md "CI Step Findings Model" owns the contract and regression list.
• ci_timeout is an idle timeout, not an absolute deadline: only timeoutAnchor re-arms when the upstream default-branch tip advances, started stays fixed for poll pacing, and re-arm only ever extends the deadline (fail-safe on transient base-tip failures). Value semantics (0 unset, negative unlimited sentinel, keyword parsing) live in config.go; keep config.DefaultCITimeout and defaultConfigYAML in sync (TestDefaultConfigYAML_MatchesGoDefaults). User-facing semantics are owned by docs/src/content/docs/reference/global-config.md.
• GitHub readiness is the union of the exact current PR head commit's check rollup and every Actions workflow run returned by the Actions API for that same SHA. A workflow rejected before creating jobs/check-runs is absent from the commit rollup but still present in that API; run discovery errors and unknown run states fail closed instead of certifying a green rollup. Regressions: TestGetChecksIncludesFailedWorkflowRunMissingFromPRRollup, TestGetChecksBindsRollupAcrossABAHeadMovement, TestCIStep_FailedHeadWorkflowRunPreventsChecksPassed.
• GitHub's raw commit statusCheckRollup returns every check run a commit ever had, including a same-named run a later run at that same head has already superseded (e.g. a required-check job re-triggered by synchronize after a pipeline auto-fix push). Host.GetChecks (internal/scm/github/github.go) collapses same-name reruns of one workflow to the newest startedAt (using Actions run identity to break timestamp ties, comparing against every retained duplicate, and preserving unordered records, including ambiguous pending replacements, before a completedAt fallback) while preserving independent workflows, unidentified external checks, and same-named commit status contexts, restoring the semantics gh pr checks already applies and this repo's own gate verify.py already assumes (last-wins). Collapse must run AFTER appendUnrepresentedWorkflowRuns, never before: that call dedupes the Actions-run union against checks by run ID over the FULL uncollapsed rollup, so collapsing first drops a superseded run's ID out of the "represented" set and the union re-adds the same stale run under its own workflow run name. Without this collapse, a monitor that keeps synchronize in its gate trigger set can loop forever: each auto-fix push legitimately fails the gate at the old head, a later same-head rebind turns it green, but the stale FAILURE stays visible and re-triggers another auto-fix round. Regressions: TestGetChecksCollapsesSupersededSameNameCheckToLatestAtOneHead, TestGetChecksCollapseOrderingDoesNotLetWorkflowRunUnionResurrectSupersededCheck, TestGetChecksPreservesIndependentSameNameWorkflows, TestGetChecksPreservesIndependentSameNameExternalCheckRuns, TestGetChecksCollapseComparesNewestRunWithEverySameNameCandidate, TestGetChecksKeepsQueuedReplacementWithEqualStartTime, TestGetChecksPreservesUnorderedExternalPendingReplacement, TestGetChecksPreservesSameNameStatusContextAndCheckRun.
• A published CI repair rebinds an existing pipeline attestation in the PR body to the new head (restampPublishedAttestation after publishRepair) only when the host has that HTML attestation and a PRContentReader (GitHub). Other hosts skip with a warning and must not fail the already-published repair. A GitHub restamp that cannot settle after retries is unsettled, not success. The CI fixer prompt may conclude that a stale/attestation/infrastructure red is not a code defect; a genuine failing test or build must still be fixed. Regressions: TestCIStep_PublishRepairRebindsAttestationAcrossRepairPushes, TestCIStep_PublishRepairDoesNotMintAttestation, TestCIStep_PublishRepairSkipsRestampWithoutReader, TestRestampPRAttestation_MissingReaderIsSkipped, TestRebindPipelineAttestationHead_VerifyPyRoundTrip, TestCIStep_FixPromptPrefersSimplificationOverMachinery, TestCIStep_FixPromptDistinguishesCodeDefectFromExternalFailure.
• CI readiness never treats an unproven empty forge check list as green. Ready requires observed all-green checks, or trusted default-branch no_ci: true with zero registered checks (internal/pipeline/steps/ci.go decides whether the declaration applies; internal/cimonitor owns the agent-facing log vocabulary and Ready/DeclaredNoCI parse). Delayed registration, pending checks, failures, errors, unknowns, and stale-head evidence stay not-ready; registered checks on a declared no-CI repo are still honored. Regressions: TestChecksPassed_PR607RealLogSequence, TestCIStep_EmptyChecksWithoutNoCIStaysNotReadyPastOldGracePeriod, TestCIStep_EmptyChecksWithTrustedNoCIBecomesReady, TestCIStep_DelayedCheckRegistrationStaysNotReadyUntilGreen, TestCIStep_DeclaredNoCIWithUnexpectedChecksHonorsThem, TestEffectiveRepoConfig_NoCITrustedOnly.
• Persistent provider check-read failures are not an invisible spin: 6 consecutive GetChecks errors while the PR is still open park the CI step at an ask-user gate whose provider-neutral finding names the provider CLI/credentials support and includes the underlying error (with a GitHub-specific clause requiring gh >= 2.50 for the gh pr checks --json call), and the streak resets on any successful read. Regression: TestCIStep_PersistentCheckReadFailureParksAtAskUser.
• Reap an orphaned monitor from outside its worktree with no-mistakes axi abort --run <id>; it needs only NM_HOME and never starts a stopped daemon. A known run succeeds only with durable terminal truth, a recorded nonterminal run fails unconfirmed, and only an unknown id is an idempotent no-op. Bare axi abort stays worktree/branch-scoped.
• A merged or closed PR observation transactionally completes an active run and its CI step; PR lifecycle state is monotonic, so duplicate or delayed observations cannot reactivate or regress a terminal run. Startup reconciles legacy pending or running rows that already hold terminal PR state before parked-run planning and generic crash recovery. Regressions: TestUpdateRunPRStateFinalizesActiveTerminalOutcomes, TestUpdateRunPRStateIgnoresDuplicateAndDelayedRegressions, TestReconcileTerminalPRRunsFinalizesLegacyActiveRows, TestRecoverOnStartup_FinalizesLegacyTerminalPRRun, e2e TestTerminalPRRunDisappearsFromActiveListing.
• A provider-reported cancelled check is never a job verdict, so the deterministic rerun runs strictly before any CI fix round: cancellation and a GitHub pre-run infrastructure failure (a job that failed in setup/action resolution before any repository step ran, flagged Check.PreRunFailure by the opt-in scm.PreRunFailureDetector and re-bucketed to cancel so it never masks a real test/lint failure, which cleared setup and failed a later step) are the outcomes that earn a rerun, either outcome after its budget parks as ask-user instead of entering the auto_fix.ci loop, and any genuine or unrecognized failure or merge conflict in the same poll suppresses reruns so real failures still escalate on their first observation. The budget is per check name per run and spent on request rather than on success, and a rerun is never issued once the published branch head no longer equals runs.head_sha, because it would certify a commit this run never delivered. Each outstanding rerun records its verified pipeline head and the same-name provider links visible when it was requested; it retires durably when the run head advances or a new conclusive non-cancel link appears. Retirement keeps the spent budget and never changes check buckets. A delayed same-named green sibling can satisfy the link trigger on the same head, matching the default branch's existing name-keyed masking; removing that limitation requires provider truth outside this policy. Classification, the deliberate TIMED_OUT/STALE exclusions, rollup-lag grace, and retirement live in internal/pipeline/steps/ci_transient.go; provider support is the optional scm.CheckRerunner (GitHub only), pre-run infrastructure detection is the optional scm.PreRunFailureDetector (GitHub only, folded in by markPreRunInfraFailures), and user-facing semantics are owned by docs/src/content/docs/reference/repo-config.md. Regressions: TestCIStep_CancelledCheckIsRerunBeforeEscalating, TestCIStep_CancelledCheckStaysUnresolvedAfterItsBudget, TestCIStep_LaggingRerunRollupKeepsWaitingForTheRepublishedCheck, TestCIStep_SameHeadGreenRerunEmitsChecksPassed, TestCIStep_DelayedSameNameCheckRetainsLegacyNameBehavior, TestCIStep_ResolvedRerunDoesNotParkALaterGreenHead, TestRetireResolvedReruns, TestRetireResolvedRerunsRetriesAfterPersistenceFailure, TestCIStep_MovedPublishedHeadTerminatesInsteadOfRerunning, TestCIStep_MovedPublishedHeadClearsCIReadiness, TestClassifyCheckFailure, TestMarkPreRunInfraFailures_RetriesInfraButNotGenuine, TestMarkPreRunInfraFailures_OptInGated, TestPreRunFailures_FlagsSetupFailureNotGenuine, TestPreRunFailures_FailsClosedOnUnreadableRun.
• Terminal is not pending. Readiness must reject every non pass/fail/skip bucket (hasUnresolvedChecks), but only checks that can still finish on their own (hasPendingChecks) may keep the monitor polling. A cancel bucket - GitHub CANCELLED, GitLab canceled, Bitbucket STOPPED, and how GitHub reports a job killed by its own timeout-minutes - is a published conclusion that nothing will replace, so with no rerun outstanding it is reported as an ask-user transient finding (unresolvedTransientFindings, selected by cancelledWithoutRerun) instead of waiting. Conflating the two is the #628 regression that hung real runs for their whole ci_timeout; an unrecognized bucket is deliberately still treated as waiting, because unknown is not evidence of terminal. Regressions: TestCIStep_CancelledCheckAmongPassingChecksEscalatesInsteadOfPollingForever, TestCIStep_ZeroRerunBudgetEscalatesCancelledCheckWithoutMakingItReady, TestCIStep_BitbucketStoppedCheckParksForADecision.
• CI readiness is read from the provider's live PR head check rollup on every poll, so it always describes the head the forge currently has for that PR; no recorded SHA gates it, and a run whose row still names a pre-advance commit must still recognize green at the head the pipeline last pushed. Regression: TestCIStep_GreenChecksAtAdvancedHeadAreRecognizedWhileRunTracksOlderHead.