Debug Interop Run

Debug Interop Run

Use this workflow for remote failures from .github/workflows/interop.yml, such as https://github.com/minhuw/coquic/actions/runs/<run-id>.

Ground Rules

• Use gh for GitHub Actions inspection.
• Use -R <owner>/<repo> whenever the URL points outside the current remote.
• Store downloaded CI material under .remote-ci/ using repository naming conventions.
• Do not commit .remote-ci/, .interop-logs/, build outputs, or other generated state.
• Use nix develop -c ... for build, test, format, lint, and local reproduction commands.
• Prefer fixing only failures that reproduce locally. Report non-reproducible remote-only failures separately.
• Before editing, inspect the dirty tree and preserve unrelated user changes.
• After fixes, validate with focused reproduction, then the relevant local tests, then nix develop -c zig build test unless there is a clear constraint.
• Commit with Conventional Commits, push to origin, and trigger .github/workflows/interop.yml manually only after validation passes.

1. Capture The Run

Parse the run URL:

run_id=<id from /actions/runs/<id>>
repo=<owner>/<repo from URL>
mkdir -p ".remote-ci/${run_id}"

Inspect run and job state:

gh run view "${run_id}" -R "${repo}" \
  --json status,conclusion,event,headSha,headBranch,url,jobs \
  > ".remote-ci/${run_id}/run.json"

gh run view "${run_id}" -R "${repo}" --log-failed \
  > ".remote-ci/${run_id}-failed.log"

If the run is still in progress, say so and inspect only completed failed jobs. If no job has failed yet, stop before downloading artifacts.

For each failed job ID, save a focused failed-job log:

gh run view --job "<job-id>" -R "${repo}" --log-failed \
  > ".remote-ci/${run_id}-job-<job-id>-failed.log"

2. Download Interop Artifacts

Download artifacts into the run directory:

mkdir -p ".remote-ci/${run_id}/artifacts"
gh run download "${run_id}" -R "${repo}" \
  --dir ".remote-ci/${run_id}/artifacts"

Relevant artifact names:

• interop-logs-<peer>: full official runner logs under .interop-logs/official.
• interop-results-<peer>: JSON result snapshot for that matrix job.
• interop-results-self: self-run result snapshot.

Use find and rg to locate:

find ".remote-ci/${run_id}/artifacts" -maxdepth 4 -type f | sort
rg -n '"result": "failed"|"result":"failed"|failed|error|timed out|Traceback' \
  ".remote-ci/${run_id}/artifacts"

3. Identify Failed Cases

For each failed job, derive:

• peer implementation, for example quic-go, picoquic, quinn, msquic, quiche, ngtcp2, neqo, mvfst, aioquic, xquic, s2n-quic, or coquic.
• direction:
  • coquic-server when result path is coquic_<peer>/results.json.
  • coquic-client when result path is <peer>_coquic/results.json.
  • both only when reproducing both directions is necessary.
• failed testcase names from results.json, including measurements such as goodput or crosstraffic if those failed.
• peer image and impl from .github/workflows/interop.yml.
• congestion-control value if the matrix or logs set COQUIC_CONGESTION_CONTROL.

Prefer the artifact results.json over prose logs for the failure list. Use logs and qlog/pcap files to explain why a testcase failed.

4. Reproduce Locally

Reproduce one peer/direction/testcase group at a time. Use the same official runner entrypoint as CI, but narrow INTEROP_TESTCASES.

Example for a peer matrix job:

rm -rf .interop-logs/official
nix develop -c env \
  INTEROP_TESTCASES='<comma-separated-failed-testcases>' \
  INTEROP_COQUIC_SERVER_TESTCASES='<same list if coquic-server is involved>' \
  INTEROP_COQUIC_CLIENT_TESTCASES='<same list if coquic-client is involved>' \
  INTEROP_PEER_IMPL='<impl from workflow>' \
  INTEROP_PEER_IMAGE='<image from workflow>' \
  INTEROP_DIRECTIONS='<coquic-server|coquic-client|both>' \
  COQUIC_CONGESTION_CONTROL='<value or empty>' \
  COQUIC_RUNTIME_TRACE=1 \
  COQUIC_PACKET_TRACE=1 \
  bash interop/run-official.sh

Example for the self job:

rm -rf .interop-logs/official
nix develop -c env \
  INTEROP_TESTCASES='<comma-separated-failed-testcases>' \
  INTEROP_PEER_IMPL='coquic' \
  INTEROP_PEER_IMAGE='coquic-interop:quictls-musl' \
  INTEROP_DIRECTIONS='coquic-server' \
  COQUIC_RUNTIME_TRACE=1 \
  COQUIC_PACKET_TRACE=1 \
  bash interop/run-official.sh

If reproduction needs analysis tooling, rely on the script's Nix fallback or enter a shell with the needed package. Keep generated .interop-logs/ out of git.

5. Diagnose

Start from the narrowed local logs:

find .interop-logs/official -type f | sort
rg -n 'failed|error|timeout|close|transport|packet|trace|qlog|result' .interop-logs/official

Inspect qlog, pcap, stdout/stderr, runner output, failed run logs, failed job logs, and result JSON for the failing testcase. Compare remote artifacts under .remote-ci/<run-id>/artifacts with local .interop-logs/official.

When the issue is QUIC protocol behavior, use the repo's QUIC RAG skill/query flow before making protocol claims. Ground conclusions in RFC citations when useful.

6. Fix And Validate

For each reproducible failure:

1. Add or update focused unit/runtime tests that fail before the fix when practical.
2. Make the smallest source change that addresses the reproduced cause.
3. Run the focused test or reproduction command.
4. Rerun the narrowed interop/run-official.sh command for the affected peer/testcase/direction.
5. Repeat until all locally reproducible failed cases pass.

Then run broader validation:

nix develop -c zig build test
git diff --check

If full validation is blocked by time or environment, state the exact constraint and run the closest focused validation instead.

7. Commit, Push, And Redispatch

Before committing:

git status --short
git diff --stat

Stage only source/test/docs changes that should be tracked. Do not stage .remote-ci/, .interop-logs/, .rag/, .zig-cache/, or downloaded/generated state.

Commit and push:

git commit -m 'fix: <concise interop failure summary>'
git push origin HEAD

Trigger a fresh Interop run:

gh workflow run .github/workflows/interop.yml --ref main
gh run list --workflow .github/workflows/interop.yml --branch main --limit 5 \
  --json databaseId,displayTitle,event,headSha,status,conclusion,url,createdAt

Report the original workflow, run ID, job, failing step, relevant log excerpt, local reproduction commands, validation commands, new commit, and new Interop run ID/URL/head SHA. Also report which remote failed cases were fixed and any failed cases that were not reproducible locally.