Debug a CI failure on an OS you are not on
You are on Linux and a test fails only on macos-latest (or you are on a Mac and it fails only on
ubuntu-latest). The slow way is a pull request per attempt: each push reruns the whole matrix,
builds everything, and takes about fifteen minutes to tell you one thing. Do not do that.
The fast way is a dispatchable workflow on your own fork, scoped to the smallest run that
answers your question. You iterate by pushing to a fork branch and re-dispatching. No pull request
is involved until you have the answer.
The two things that make it fast
Scope the run. Do not run the matrix, the lint job, and the whole suite to learn one test.
Run one OS, no lint, and one test. full-tests.yml takes three inputs for exactly this:
gh workflow run full-tests.yml -f ref=<branch|sha> -f os=macos-latest -f lint=false -f filter=<substring>
os picks a single runner (macos-latest, ubuntu-latest, or both).
lint=false skips fmt and clippy.
filter is a substring of the cargo test name (not the test-binary name). It runs only the
tests whose name matches.
- Build time does not shrink with
filter (the build is --all-targets for the trampoline). The
saving is one OS instead of two, no lint, and far fewer tests that execute. To also cut build
time, scope the build in a throwaway workflow with --test <binary> or -p <crate>.
For an environment question, build nothing. Most "green here, red there" failures are the
environment, not the code: a different active toolchain, a different system path, a pointer file,
a library location. A workflow that only runs shell commands returns in about fifteen seconds, so
you can ask five questions in the time one build takes. Print what you suspect differs:
- macOS:
sw_vers, xcode-select -p, xcrun --find <tool>, which -a <tool>, otool -L <bin>,
readlink of any pointer files, the relevant env vars, and for an interpreter
python3 -c 'import sys,os; print(sys.executable, os.path.realpath(sys.executable))'.
- Linux:
uname -a, the distribution release, which -a <tool>, readlink -f <bin>, ldd <bin>,
and the mount or namespace state that matters.
Two fork rules, or the dispatch fails
- A pull request from a fork runs the base repository's workflow, not your fork's copy. So a
workflow edit on a fork branch does nothing on a PR. It only takes effect through a manual
dispatch on the fork (or after it merges to the base).
workflow_dispatch needs the workflow file on the fork's default branch to be dispatchable at
all. Push the workflow to the fork default branch once; after that you can dispatch it against any
branch or sha with -f ref=....
The loop
- Confirm the divergence: check out the exact CI commit, run the test locally, see it pass. Now you
know the code is fine and the environment differs. Stop reading the diff.
- Push a no-build probe workflow to your fork default branch. Dispatch it. Read what the runner's
environment holds that yours does not.
- Form one hypothesis. Test it with the smallest scoped run: one OS, no lint, one filtered test, or
the probe with one more command. Change one thing per run.
- Read the result.
gh run watch <id>, or gh api repos/<owner>/<fork>/actions/jobs/<jobid>/logs
for the raw log. The runner's own log names the cause; do not infer it from a test timeout.
- When a run confirms the fix, land it and do one final scoped dispatch to prove it green.
When the failure is containment (deny-default box)
If the failing test starts a contained workload and it dies at startup (a leaf launch, a Seatbelt or
namespace refusal, a startup-marker timeout), the environment difference is usually a path the box's
floor does not grant on the runner. Two extra moves apply, and the full method plus a worked example
(the macOS /usr/bin/python3 xcode-select leaf) live in
docs/dev/debug-os-specific-ci-failures.md:
- Reproduce the box's floor by hand. On macOS write a
sandbox-exec profile from the agent and
leaf cells in crates/containment/src/containment-data/os-paths.macos.json; on Linux
reconstruct the launcher's binds or run the trampoline directly. Validate the reconstruction
passes locally before you trust its runner failure.
- Read the kernel's own denial, not the test's timeout: on macOS
sudo log show --style syslog --last 3m --predicate 'eventMessage CONTAINS "deny"'; on Linux
dmesg, journalctl, or an strace of the refused syscall.
1---2name: debug-os-failure-on-github3description: "Debug a CI failure on an OS you are not on (you are on Linux, it fails on macos-latest, or the reverse) without opening a pull request per attempt. Use when a test is green on your machine and red on a GitHub runner, or when you need to run one test or one shell probe on a runner and read the result fast. The core move is a dispatchable, scoped workflow on your own fork: pick one OS, skip lint, filter to one test, and for environment questions run a no-build probe that returns in seconds instead of a ten-minute build. Iterate by pushing to the fork branch and re-dispatching, never by editing a PR."4metadata:5 source: https://skillmd.com/skills/strands-agents/debug-os-failure-on-github6---78# Debug a CI failure on an OS you are not on910You are on Linux and a test fails only on `macos-latest` (or you are on a Mac and it fails only on11`ubuntu-latest`). The slow way is a pull request per attempt: each push reruns the whole matrix,12builds everything, and takes about fifteen minutes to tell you one thing. Do not do that.1314The fast way is a **dispatchable workflow on your own fork**, scoped to the smallest run that15answers your question. You iterate by pushing to a fork branch and re-dispatching. No pull request16is involved until you have the answer.1718## The two things that make it fast19201. **Scope the run.** Do not run the matrix, the lint job, and the whole suite to learn one test.21 Run one OS, no lint, and one test. `full-tests.yml` takes three inputs for exactly this:2223 ```sh24 gh workflow run full-tests.yml -f ref=<branch|sha> -f os=macos-latest -f lint=false -f filter=<substring>25 ```2627 - `os` picks a single runner (`macos-latest`, `ubuntu-latest`, or `both`).28 - `lint=false` skips fmt and clippy.29 - `filter` is a substring of the cargo **test name** (not the test-binary name). It runs only the30 tests whose name matches.31 - Build time does not shrink with `filter` (the build is `--all-targets` for the trampoline). The32 saving is one OS instead of two, no lint, and far fewer tests that execute. To also cut build33 time, scope the build in a throwaway workflow with `--test <binary>` or `-p <crate>`.34352. **For an environment question, build nothing.** Most "green here, red there" failures are the36 environment, not the code: a different active toolchain, a different system path, a pointer file,37 a library location. A workflow that only runs shell commands returns in about fifteen seconds, so38 you can ask five questions in the time one build takes. Print what you suspect differs:3940 - macOS: `sw_vers`, `xcode-select -p`, `xcrun --find <tool>`, `which -a <tool>`, `otool -L <bin>`,41 `readlink` of any pointer files, the relevant env vars, and for an interpreter42 `python3 -c 'import sys,os; print(sys.executable, os.path.realpath(sys.executable))'`.43 - Linux: `uname -a`, the distribution release, `which -a <tool>`, `readlink -f <bin>`, `ldd <bin>`,44 and the mount or namespace state that matters.4546## Two fork rules, or the dispatch fails4748- A pull request from a fork runs the **base** repository's workflow, not your fork's copy. So a49 workflow edit on a fork branch does nothing on a PR. It only takes effect through a manual50 dispatch on the fork (or after it merges to the base).51- `workflow_dispatch` needs the workflow file on the fork's **default** branch to be dispatchable at52 all. Push the workflow to the fork default branch once; after that you can dispatch it against any53 branch or sha with `-f ref=...`.5455## The loop56571. Confirm the divergence: check out the exact CI commit, run the test locally, see it pass. Now you58 know the code is fine and the environment differs. Stop reading the diff.592. Push a no-build probe workflow to your fork default branch. Dispatch it. Read what the runner's60 environment holds that yours does not.613. Form one hypothesis. Test it with the smallest scoped run: one OS, no lint, one filtered test, or62 the probe with one more command. Change **one** thing per run.634. Read the result. `gh run watch <id>`, or `gh api repos/<owner>/<fork>/actions/jobs/<jobid>/logs`64 for the raw log. The runner's own log names the cause; do not infer it from a test timeout.655. When a run confirms the fix, land it and do one final scoped dispatch to prove it green.6667## When the failure is containment (deny-default box)6869If the failing test starts a contained workload and it dies at startup (a leaf launch, a Seatbelt or70namespace refusal, a startup-marker timeout), the environment difference is usually a path the box's71floor does not grant on the runner. Two extra moves apply, and the full method plus a worked example72(the macOS `/usr/bin/python3` xcode-select leaf) live in73[docs/dev/debug-os-specific-ci-failures.md](../../../docs/dev/debug-os-specific-ci-failures.md):7475- Reproduce the box's floor by hand. On macOS write a `sandbox-exec` profile from the `agent` and76 `leaf` cells in `crates/containment/src/containment-data/os-paths.macos.json`; on Linux77 reconstruct the launcher's binds or run the trampoline directly. Validate the reconstruction78 passes locally before you trust its runner failure.79- Read the kernel's own denial, not the test's timeout: on macOS80 `sudo log show --style syslog --last 3m --predicate 'eventMessage CONTAINS "deny"'`; on Linux81 `dmesg`, `journalctl`, or an `strace` of the refused syscall.