Publish Health Check

Publish health check

Notices, without anyone manually querying the Actions API, when a Skillz Forge publish silently stops going live.

Why this exists

.github/workflows/deploy-pages.yml once failed on every push for months after a monorepo migration, and nobody noticed: pushes to main looked normal, but the live site quietly served a stale build. There was no signal that surfaced the failure without an agent deliberately going and checking.

This check closes that gap two ways at once:

1. It re-checks the last deploy-pages.yml run's conclusion for main, so a failed publish is flagged even if nobody is watching the Actions tab.
2. It independently compares the live catalog.json's sourceCommit against the commit that the last successful deploy run actually built (its head_sha) -- not against main's current tip, since deploy-pages.yml only fires on a push.paths allowlist and plenty of legitimate main commits (docs, unrelated workflows) never trigger a deploy at all. A mismatch here means a run reported "success" but didn't actually ship the commit it built (a stale Pages cache, a mis-scoped artifact, etc.) -- reporting success is not the same as being live.

Running it

node .agents/skills/publish-health-check/run.mjs
node .agents/skills/publish-health-check/run.mjs --json validation-reports/publish-health-check.json

No credentials are required -- it only issues public, unauthenticated fetch calls against the GitHub REST API (workflow run history) and the published GitHub Pages site.

Regression check: a main commit that does not touch any of deploy-pages.yml's watched paths never triggers a deploy, so it must not be misreported as a stale site. This was verified by pushing a change to the top-level README (which matches none of deploy-pages.yml's push.paths patterns -- unlike this SKILL.md, which does) and confirming the check still reported healthy, correctly comparing against the previous successful run's head_sha instead of main's new tip.

Exit codes distinguish why it's unhealthy, since each needs a different response:

CI wiring

.github/workflows/publish-health-check.yml runs this on a schedule (every 30 minutes) and immediately after every deploy-pages.yml run completes, so a failure surfaces within minutes rather than at the next unrelated push. An scheduled unhealthy result must persist across three consecutive scheduled runs before it opens or updates a tracking issue, preventing a one-off CDN, DNS, or API blip from becoming an incident. A workflow_run check caused by a failed deploy bypasses that debounce and alerts immediately. Healthy checks reset the scheduled streak, and a return to healthy automatically closes an existing issue. See that workflow file for the exact issue-management steps.

The streak is persisted in the replaceable publish-health-streak Actions artifact and protected by the workflow's concurrency group, so separate scheduled runs share the same count without committing generated state.

If the repository secret PUBLISH_HEALTH_WEBHOOK_URL is configured, the same workflow sends a real-time transition notification to that webhook when an incident is first opened and when it recovers. It accepts Slack incoming webhooks and Discord webhooks: the workflow sends both text and content fields so either service can consume the message. Repeated checks during the same incident do not send another webhook notification. If the secret is absent, the notification step is skipped and GitHub issue tracking continues normally.

The notification sender reports the provider's HTTP rejection (without printing the secret URL), writes a redacted failure record for the workflow, and the explicit publish-health failure step still runs afterward, so a delivery problem cannot obscure the underlying check failure. A separate publish-health-webhook tracking issue is opened or updated when delivery fails; it remains open independently of the publish-health incident issue, so a failed recovery notification remains discoverable after the incident closes. Run the local contract fixture to verify the failure, debounce, recovery, and rejection paths without credentials:

node .agents/skills/publish-health-check/webhook-test.mjs

Run the overlapping-trigger contract fixture to verify that scheduled failures remain debounced across successful workflow_run catch-up checks, failed deploys alert immediately, the third scheduled failure opens only one incident, and the streak artifact state is replaced safely:

node .agents/skills/publish-health-check/debounce-test.mjs