Reproduce — Verify the Bug Exists
Runs a test and determines whether the bug reproduces. Uses a three-stage approach: nightly wheel first (fast), source build at CI commit second (precise), CI environment alignment third (last resort).
The orchestrator decides what to do with the output — this skill only reports the result.
Inputs
reproducer_command— the sequence of shell commands that triggers the failure. Any of the forms below is valid; Stage 1's "Reproducer forms" section routes execution:- A pytest node id or
pytest ...invocation:pytest -v test/xpu/test_ops.py::TestFooXPU::test_bar_xpu_float32 - A
python -c "..."snippet or a singlepython script.pyline. - A multi-line shell block (as is common in issue bodies): env setup,
git clone,pip install, followed by the actual failing command.
Missing or unrunnable inputs return
NO_REPRODUCERup-front — see "## Preflight" below. Non-NO_REPRODUCERverdicts come from the Stage 1/2/3 execution flow.Providers (set by the orchestrator, not this skill):
- Issue body — extracted by
issue-triagefrom the reproducer section (viaissue-handler). - CI failure log — the failing pytest node id from the nightly CI report, passed through as a batch sub-item.
- A pytest node id or
stage— which reproduction path to run. Defaultauto.auto— run the full three-stage fallback chain (nightly → source_build → ci_env). Used by orchestrators that need a definitive verdict.nightly— only run Stage 1 (nightly wheel). PASS returnsNOT_REPRODUCED(checked_stages=[nightly])immediately; do NOT fall through to source build or ci_env. Cheapest option — suitable for a fast "does this still reproduce on latest nightly?" answer.
ci_commit— upstream commit hash from the CI report. Only used as a fallback base whenorigin/mainfails to build (optional; ignored whenstage=nightly).pytorch_dir— path to a local PyTorch checkout (optional). Used whenever Prepare determinesneeds_tree=yes(any pytest form with a repo-relative path) as well as by Stage 2's source build. If absent, clone to<torch-xpu-ops-repo-root>/agent_space_xpu/pytorch/(agent_space_xpu/is the gitignored scratch dir at the torch-xpu-ops repo root — see the containing repo'sAGENTS.md).ci_repo— which CI to align against in Stage 3:pytorchortorch-xpu-ops. Optional; when absent, Stage 3 infers from the reproducer path (see "Determineci_repo" in Stage 3).
Preflight
NO_REPRODUCER is a pre-execution verdict — the skill decides that
there is nothing to run before touching any stage. Every other verdict
(REPRODUCED / NOT_REPRODUCED / CANNOT_VERIFY) comes from Stage 1/2/3
execution.
reproducer_command present? If missing or empty:
NO_REPRODUCER(reason=no_command). Stop.
If the input is a stack trace without a command, the orchestrator should
not call this skill in the first place (that is issue-triage's
reproduction_missing=yes case); if it slips through, this check
catches it.
The pytest collected 0 items check happens in Prepare below, after
the source tree it needs to run against is in place.
Prepare
Some reproducer forms need a pytorch source tree even at Stage 1
(nightly wheel path) — either because the test file lives inside
pytorch/test/ or because it lives under torch-xpu-ops/test/xpu/
and imports common test utilities via
sys.path.append("../../../../test/functorch") relative paths that
only resolve from <pytorch_dir>/third_party/torch-xpu-ops/test/xpu/.
When to prepare
Set needs_tree from the reproducer form:
| Reproducer form | needs_tree |
|---|---|
pytest, path is repo-relative (test/xpu/..., test/...) |
yes |
pytest, bare node id without a file path (TestFoo::test_bar) |
yes — pytest rootdir discovery needs the tree |
| pytest, path is absolute and exists on disk | no |
python -c "..." / python /abs/path/script.py |
no |
| shell block (issue body: clone + install + run) | no — the block does its own setup |
If needs_tree=no, skip this section and go to Stage 1.
Get the pytorch tree
If pytorch_dir was provided as input: git -C $pytorch_dir fetch origin.
If not provided, clone into the torch-xpu-ops repo's gitignored scratch dir. Resolve the path explicitly rather than relying on cwd:
XPU_OPS_ROOT=$(git -C <path-to-torch-xpu-ops-checkout> rev-parse --show-toplevel)
pytorch_dir="$XPU_OPS_ROOT/agent_space_xpu/pytorch"
if [[ ! -d "$pytorch_dir/.git" ]]; then
git clone --filter=blob:none https://github.com/pytorch/pytorch.git "$pytorch_dir"
fi
git -C "$pytorch_dir" fetch origin
git -C "$pytorch_dir" checkout --detach origin/main
git -C "$pytorch_dir" submodule update --init --recursive
The tree is not built here — Stage 1 uses the nightly wheel for the runtime; the source tree only supplies test files and support modules. Stage 2 reuses the same tree and builds it there.
torch-xpu-ops test path
If the reproducer targets test/xpu/..., make the working torch-xpu-ops
tree available at $pytorch_dir/third_party/torch-xpu-ops. The build's
dev-override recipe (symlink or replace-clone) applies; see
xpu-build-pytorch. From here on, Stage 1's cwd for pytest is
$pytorch_dir/third_party/torch-xpu-ops/test/xpu/.
Collect-only check (pytest form)
Regardless of needs_tree, if reproducer_command matches the pytest
form (starts with pytest, python -m pytest, or is a bare pytest
node id), run:
pytest --collect-only <node_id>
Cwd:
needs_tree=yes→ from the tree just prepared (fortest/xpu/...targets, that's$pytorch_dir/third_party/torch-xpu-ops/test/xpu/)needs_tree=no→ from any non-pytorch directory (the reproducer's absolute path resolves on its own)
Output shows collected 0 items? → NO_REPRODUCER(reason=collected_zero).
Stop. Do not fall through to Stage 2 — the source tree is the same
across stages, so a collect-miss at Stage 1 will miss at 2 and 3 too.
Non-pytest forms have no equivalent pre-execution check.
Stage 1: Nightly Wheel (fast path)
Most failures reproduce here. Start here before doing anything heavier.
Reproducer forms
Three forms; each dispatches differently in "Run test" below:
- Pytest form —
reproducer_commandstarts withpytest,python -m pytest, or is a bare pytest node id (.../test_foo.py::TestBar::test_baz). Thecollect-onlycheck (see "## Prepare" above) has already run. - Python one-liner / single-script form —
python -c "..."orpython path/to/script.py. - Shell-block form — a multi-line block copied out of an issue
body: env setup +
git clone+pip install+ the failing command. Split it into setup steps (everything before the failing command) and the reproduce step (the last command that exercises the failing path). Only the reproduce step's outcome determines the verdict; setup-step failures returnCANNOT_VERIFY(stage=<current>, blocker=<the failing setup step>).
The Working directory and Use the test's own assertion rules below apply to all three forms.
Install
Always reproduce against the latest available XPU nightly. Do not reuse a
stale wheel from a previous session — a bug may already be fixed in a newer
nightly, and re-verifying an old wheel produces misleading REPRODUCED
results.
# Query available versions (informational — pip install --upgrade below
# will pick a resolvable one, which may lag the newest entry here by a
# day when the index metadata refreshes before all wheels land).
pip3 index versions torch --pre \
--index-url https://download.pytorch.org/whl/nightly/xpu
pip3 install --pre --upgrade torch torchvision torchaudio \
--index-url https://download.pytorch.org/whl/nightly/xpu
Post-install, check that torch, torchvision, torchaudio are all
from the same day — pip's resolver can leave a mixed set (either torch
older than the auxiliary wheels, or the reverse). If they diverge,
uninstall all three and reinstall together:
pip3 uninstall -y torch torchvision torchaudio
pip3 install --pre torch torchvision torchaudio \
--index-url https://download.pytorch.org/whl/nightly/xpu
Record the exact wheel version used (python -c "import torch; print(torch.__version__)") in the reproduce output and in any issue comment,
so downstream stages and re-verifications know which nightly was tested.
Working directory
Do NOT run the nightly-wheel reproducer with cwd inside any pytorch
source checkout. Python resolves import torch against the local torch/
package before site-packages, so it will load the in-tree torch/_C.so
built at whatever revision that tree happens to be — typically stale
relative to the installed wheel — and fail with
ImportError: undefined symbol: .... Either cd $(mktemp -d) (or any
non-pytorch dir) before running, or invoke the reproducer with an
absolute path from outside the tree.
torch-xpu-ops test invocation
Applies when the reproducer targets a test under torch-xpu-ops/test/xpu/.
Prepare has already ensured the pytorch tree exists at $pytorch_dir
with the working torch-xpu-ops tree at
$pytorch_dir/third_party/torch-xpu-ops. Invoke the reproducer from
$pytorch_dir/third_party/torch-xpu-ops/test/xpu/ — the relative
sys.path.append("../../../../test/functorch") in those tests only
resolves from that cwd.
The pytorch tree does NOT need to be built for the nightly-wheel path — the wheel provides the runtime, the source tree only supplies test files and support modules.
Use the test's own assertion
When writing a standalone reproducer for a TestCase.assertEqual
failure, use the test's own assertion. Do NOT substitute
torch.allclose, torch.equal, or bare == — they have different
(usually stricter) default tolerances and will manufacture false
positives.
If the failure log says AssertionError: Tensor-likes are not close,
the assertion is torch.testing._comparison.assert_close, which has
dtype-specific defaults (bf16: rtol=0.016, atol=1e-5). Reproduce
through assert_close or via TestCase.assertEqual:
import sys; sys.path.insert(0, "<pytorch>/test")
from torch._dynamo.test_case import TestCase # or the base class the failing test uses
class T(TestCase):
def test_x(self, device):
...
self.assertEqual(out_ref, out)
T().test_x(device='xpu')
Run test
Run according to the reproducer form matched in "Reproducer forms":
Pytest form: run the pytest invocation. Result interpretation:
FAILED→ REPRODUCED.all skippedby@skipIfXpu→ the marker is likely hiding the actual failure. Temporarily remove it in place, re-run once to check what happens without the skip, then revert the file so no change escapes this skill:# Remove @skipIfXpu from the target test file(s), then: pytest <node_id> # Regardless of outcome, revert: git checkout <test_file>If the re-run FAILs → REPRODUCED (the skip was hiding it). If it PASSes → treat as PASSED per the Decision table. Only return
CANNOT_VERIFYwhen the skip is environmental (not an XPU marker, e.g.@skipIf(not has_cuda)shielding an unavailable dep).xfailed→ treat asFAILED(REPRODUCED).PASSED→ per the Decision table below.
Python one-liner / shell block: run the command (or the reproduce step extracted from the shell block, per "Reproducer forms"). Result interpretation:
- Exit code non-zero and output matches the failure pattern named in the issue (traceback, error message, or specific assertion) → REPRODUCED.
- Exit code zero → PASSED (per the Decision table below).
- Exit code non-zero but cause is unrelated (missing dependency
surfacing inside the reproduce step, permission error, missing
device) →
CANNOT_VERIFY(stage=nightly, blocker=<...>). Do NOT report REPRODUCED on a setup or infra failure.
all skipped/xfaileddo not apply to these forms — they are pytest-specific concepts.
Decision
| Result | Condition | Action |
|---|---|---|
CANNOT_VERIFY |
env problem (wheel install failed, runtime missing) | Report to orchestrator, stop |
REPRODUCED |
FAILED | Return REPRODUCED(stage=nightly, refined_command=...) |
| → stage 2 | PASSED and stage=auto |
Proceed to source build at origin/main to confirm |
NOT_REPRODUCED |
PASSED and stage=nightly |
Return NOT_REPRODUCED(checked_stages=[nightly]) — do NOT fall through |
Stage 2: Source Build at origin/main
Nightly passing is not conclusive — it may lag behind CI. Build from
origin/main to verify. Even when the failure came from a specific CI
commit, we only consider fixes on top of origin/main — downstream
stages branch off it.
Prepare pytorch checkout
If Prepare (above) already ran (needs_tree=yes), $pytorch_dir is
detached at origin/main with submodules initialized — skip to
"Build and run".
Otherwise (Prepare was skipped because needs_tree=no), run Prepare's
"Get the pytorch tree" recipe now: resolve $pytorch_dir (from input
or $XPU_OPS_ROOT/agent_space_xpu/pytorch), clone if missing, fetch,
checkout --detach origin/main, submodule update --init --recursive.
Leave HEAD detached at origin/main at exit (the ci_commit fallback
below re-detaches to a different sha; downstream stages branch off
whatever this stage settled on).
Build and run
Load the xpu-build-pytorch skill and follow it for the build. Do not
hand-roll the build here.
If the origin/main build fails for a reason unrelated to the bug
(broken trunk, upstream infra issue, etc.) and ci_commit is
available, fall back once:
git -C $pytorch_dir checkout --detach $ci_commit
git -C $pytorch_dir submodule update --init --recursive
Rebuild via xpu-build-pytorch. If this succeeds, proceed with the
test on ci_commit and record base=<ci_commit_sha> in the output so
the orchestrator branches its fix off the same base. If the fallback
build also fails, escalate as
CANNOT_VERIFY(blocker=trunk and ci_commit both fail to build)
rather than silently reproducing on some other base.
Then run the reproducer following the form-specific rules in Stage 1 "Run test" (pytest interpretation vs python/shell interpretation).
Decision
Applies only when stage=auto — stage=nightly returns at Stage 1.
| Result | Action |
|---|---|
CANNOT_VERIFY |
Report to orchestrator, stop |
REPRODUCED |
Return REPRODUCED(stage=source_build, base=origin/main|<ci_commit_sha>, refined_command=...) |
PASSED |
Proceed to stage 3 |
Stage 3: CI Environment Alignment
Only reached when nightly wheel and source build at origin/main both
pass. The failure may be specific to the CI environment: wheels built
under CI toolchain, XPU/oneAPI stack pinned to a specific version,
environment variables set by CI.
Assumption: the agent already runs inside the CI test container
Both pytorch/pytorch and torch-xpu-ops run their XPU tests inside
a container (declared as container: image: in the workflow yaml).
When this skill is invoked from within CI (e.g. via @torchxpubot fix), the agent is already inside that container — kernel modules,
/dev/dri, oneAPI stack, and python are already the CI ones. This
stage does not docker pull or docker run. It aligns the
installed wheels + pytorch source checkout to the CI wheel, then runs
the reproducer directly.
The skill logs the CI image reference it identified (for context in the report), but does not exec into it.
Clean up Stage 2 artifacts
Stage 2 may have left behind build/, torch/lib/*.so, or a modified
third_party/xpu.txt from the dev-override. Left in place, Python
will pick the host-built (stale-relative-to-CI-wheel) torch/_C.so
off sys.path and error with undefined symbol before the
reproducer runs.
# Restore xpu.txt to origin's pinned commit (in case Stage 2 rewrote it).
git -C "$pytorch_dir" checkout -- third_party/xpu.txt
# Discard stage-2 build outputs. `git clean` does not recurse into
# nested repositories by default, so third_party/torch-xpu-ops (a
# separate git repo) is preserved without needing `-e`.
git -C "$pytorch_dir" clean -fdx
Alternatively, run the reproducer with cwd outside $pytorch_dir
(e.g. cd /tmp) so import torch resolves against site-packages,
matching Stage 1's "Working directory" rule. Do at least one.
Determine ci_repo
Pick which CI to align against based on the reproducer:
| Reproducer clue | ci_repo |
|---|---|
Path contains test/xpu/ or torch-xpu-ops |
torch-xpu-ops |
Path is pytorch/test/... or absolute path inside a pytorch tree |
pytorch |
Ambiguous / python -c snippet with no path |
try torch-xpu-ops first, fall back to pytorch |
The orchestrator may also pass ci_repo explicitly; when set, use it
and skip the heuristic.
Path A — ci_repo=torch-xpu-ops
Wheels come from intel/torch-xpu-ops's own build workflow and stay on
GitHub Actions artifact storage. Fetch via gh run download, not S3.
A1. Find the latest successful wheel-producing run
The build job lives in _linux_build.yml (a reusable workflow called
by pull.yml and nightly_ondemand.yml). It uploads the artifact
Torch-XPU-Wheel-<pr|sha>-<runid>-<attempt>[-category].
# Nightly is the primary source (fresh main-branch build every night).
# Fall back to pull.yml when nightly has been failing for a stretch —
# pull.yml runs on PRs against main and its wheels are close enough for
# CI-env alignment.
RUN=$(gh run list --repo intel/torch-xpu-ops \
--workflow nightly_ondemand.yml \
--status success --limit 1 \
--json databaseId,headSha,createdAt)
if [[ "$RUN" == "[]" ]]; then
RUN=$(gh run list --repo intel/torch-xpu-ops \
--workflow pull.yml \
--status success --limit 1 \
--json databaseId,headSha,createdAt)
fi
# Empty here → report CANNOT_VERIFY per the paragraph below (do not
# `exit`; the skill returns a verdict, it does not terminate the shell).
RUN_ID=$(jq -r '.[0].databaseId' <<<"$RUN")
If both queries return empty: CANNOT_VERIFY(stage=ci_env, blocker=no_recent_successful_torch-xpu-ops_run).
A2. Download the wheel artifact
CI_ENV_DIR="$XPU_OPS_ROOT/agent_space_xpu/ci_env"
WHEELS_DIR="$CI_ENV_DIR/wheels"
rm -rf "$WHEELS_DIR" && mkdir -p "$WHEELS_DIR"
# Artifact name is Torch-XPU-Wheel-<pr|sha>-<runid>-<attempt>[-category].
# `gh run download -n <name>` requires exact name; use pattern instead.
# If the run uploaded multiple category variants (target/baseline via
# `_linux_build.yml`'s `category` input), --pattern pulls all of them
# and the flatten below will clobber same-named wheels. In that case
# pass --name <specific-artifact> to pick one variant.
gh run download "$RUN_ID" --repo intel/torch-xpu-ops \
--pattern 'Torch-XPU-Wheel-*' --dir "$WHEELS_DIR"
# gh unpacks each artifact into its own subdir; flatten:
find "$WHEELS_DIR" -mindepth 2 -name '*.whl' -exec mv {} "$WHEELS_DIR" \;
# Empty here → CANNOT_VERIFY(stage=ci_env, blocker=no_wheel_in_artifact).
# Do not `exit`; return the verdict via the skill's Output section.
find "$WHEELS_DIR" -maxdepth 1 -name '*.whl' | grep -q .
A3. CI image (reference only)
For torch-xpu-ops the test container is
intelgpu/ubuntu-24.04-lts2:2523.40 (see
.github/workflows/_linux_ut.yml). Record the tag for the report;
do not pull. Kept for local-investigation convenience (someone
reproducing outside CI can docker run this image manually) — the
skill itself relies on the Assumption above.
Path B — ci_repo=pytorch
Wheels come from pytorch/pytorch's xpu workflow and land on
gha-artifacts S3.
B1. Find the latest successful xpu workflow run
Accept only runs where every linux-*/ build job succeeded — partial
runs still upload partial artifacts.
# Match by display name first; fall back to path in case pytorch/pytorch
# renames the workflow's `name:` field (the file path is more stable).
WF_ID=$(gh api "repos/pytorch/pytorch/actions/workflows?per_page=100" --paginate \
--jq '.workflows[] | select(.name=="xpu" or .path==".github/workflows/xpu.yml") | .id' \
| head -1)
RUN_ID=""
for page in 1 2 3 4 5; do
while IFS=$'\t' read -r rid _ _; do
conclusions=$(gh api \
"repos/pytorch/pytorch/actions/runs/$rid/jobs?per_page=100" --paginate \
--jq '.jobs[] | select(.name | test("^linux.*/ build$")) | .conclusion')
[[ -z "$conclusions" ]] && continue
grep -qv '^success$' <<<"$conclusions" && continue
RUN_ID=$rid; break 2
done < <(gh api \
"repos/pytorch/pytorch/actions/workflows/$WF_ID/runs?status=completed&per_page=20&page=$page" \
--jq '.workflow_runs[] | [.id, .head_sha, .created_at] | @tsv')
done
If no qualifying run in the last 100: CANNOT_VERIFY(stage=ci_env, blocker=no_recent_successful_xpu_workflow_run).
B2. Pick the right build_env
Per the Assumption above, the agent is already inside a compatible CI container — the image column below is a lookup for the report only, not a pull target. It is kept in case a local investigation (outside CI) wants to spin up the same image manually to reproduce; the skill itself does not use it.
pytorch/pytorch's xpu.yml currently defines only py3.10 linux builds:
| build_env | Hardware | Image (reference) |
|---|---|---|
linux-noble-xpu-n-py3.10 |
PVC | ghcr.io/pytorch/ci-image:pytorch-linux-noble-xpu-n-py3-<docker-tree-hash> |
linux-noble-xpu-n-py3.10-client |
BMG | ghcr.io/pytorch/ci-image:pytorch-linux-noble-xpu-n-py3-client-<docker-tree-hash> |
linux-jammy-xpu-n-1-py3.10 |
PVC | ghcr.io/pytorch/ci-image:pytorch-linux-jammy-xpu-n-1-py3-<docker-tree-hash> |
-client suffix = BMG (client GPU); no suffix = PVC (datacenter).
Match the runner's hardware; default to PVC when unknown.
If xpu.yml grows a new build_env not covered here:
CANNOT_VERIFY(stage=ci_env, blocker=unknown_build_env=<name>). Do
not guess. When multiple envs match, iterate them in sorted order for
determinism.
<docker-tree-hash> is the git tree hash of .ci/docker/ at the run's
commit (see upstream _runner-determinator.yml "Compute .ci/docker
tree hash"). Only needed if the report wants a fully-qualified image
reference; skill does not pull the image.
B3. Download wheel artifacts
Artifacts live at:
https://gha-artifacts.s3.amazonaws.com/pytorch/pytorch/<run_id>/<build_env>/artifacts.zip
Probe availability with --range 0-0 -L (zero-byte GET); HEAD may be
rejected by some intermediaries in front of this bucket, byte-range
GET returns 200 or 206:
url="https://gha-artifacts.s3.amazonaws.com/pytorch/pytorch/$RUN_ID/$BUILD_ENV/artifacts.zip"
http_status=$(curl -s -o /dev/null -w "%{http_code}" --range 0-0 -L "$url")
# 200 or 206 -> ok; anything else -> skip this build_env
Download and extract:
CI_ENV_DIR="$XPU_OPS_ROOT/agent_space_xpu/ci_env"
WHEELS_DIR="$CI_ENV_DIR/wheels"
ARTIFACTS_ZIP="$CI_ENV_DIR/artifacts.zip"
rm -rf "$WHEELS_DIR" && mkdir -p "$WHEELS_DIR"
# --retry survives transient drops. A silent truncation of the 1.2 GB
# zip surfaces later as "cannot find zipfile directory" from unzip.
curl -sL -f --retry 3 --retry-delay 5 "$url" -o "$ARTIFACTS_ZIP"
# Layout varies: some build envs pack wheels under dist/, others at root.
unzip -o -j "$ARTIFACTS_ZIP" 'dist/*.whl' -d "$WHEELS_DIR" \
|| unzip -o -j "$ARTIFACTS_ZIP" '*.whl' -d "$WHEELS_DIR"
# Empty here → CANNOT_VERIFY(stage=ci_env, blocker=no_wheel_extracted).
find "$WHEELS_DIR" -maxdepth 1 -name '*.whl' | grep -q .
Install the CI wheel and align source
Same for both paths. Uninstall any existing torch stack first — the Stage 1 nightly is still resident:
pip uninstall -y torch torchvision torchaudio pytorch-triton-xpu triton_xpu 2>/dev/null || true
pip install --force-reinstall "$WHEELS_DIR"/*.whl
Align the pytorch source tree to the wheel's commit so tests that
import support modules from pytorch/test/ see matching code (Prepare
left $pytorch_dir detached at origin/main, which is not the wheel's
commit):
TORCH_COMMIT_ID=$(python -c 'import torch; print(torch.version.git_version)')
git -C "$pytorch_dir" fetch origin
git -C "$pytorch_dir" checkout --detach "$TORCH_COMMIT_ID"
git -C "$pytorch_dir" submodule update --init --recursive
Do not shallow-fetch (--depth 1) here — pytorch's nested submodules
resolve against pins that require the full history to be reachable;
a shallow fetch surfaces later as "cannot find " in submodule update.
TORCH_COMMIT_ID (the wheel's build commit) is a temporary
alignment only — it makes pytorch/test/ support modules match the
installed wheel's binary so the test can run. It is not the fix
base and is never returned as base. Downstream fixes always branch
off origin/main (see Stage 2), so this stage still reports
base=origin/main; TORCH_COMMIT_ID stays internal to Stage 3.
For torch-xpu-ops test paths, ensure the working torch-xpu-ops tree is
at $pytorch_dir/third_party/torch-xpu-ops (Prepare already handled
this if needs_tree=yes; if it didn't, do it now via the
xpu-build-pytorch dev-override recipe).
Run the reproducer
Run in the current shell — no docker run wrapper. Working-directory
rule from Stage 1 applies: for torch-xpu-ops tests use
$pytorch_dir/third_party/torch-xpu-ops/test/xpu/; for a
non-test-file reproducer, cd /tmp (or any non-pytorch dir).
Result interpretation is the form-specific rule from Stage 1 "Run test" — pytest FAILED / all skipped / xfailed, or non-pytest exit code
- output-vs-failure-pattern.
What to check if the failure still doesn't reproduce
From the CI job log, extract and align remaining differences:
- Full test command with all flags (
--timeout,-x, specific env vars) - Any environment variables set in the CI job (
ZE_AFFINITY_MASK,PYTORCH_TEST_WITH_XPU,IS_XPU_CI, etc.)
When aligning yields REPRODUCED, fold discovered pieces (env vars,
flags, cwd) into refined_command per its contract in the Output
section.
Decision
| Result | Action |
|---|---|
CANNOT_VERIFY |
Report to orchestrator, stop |
REPRODUCED |
Return REPRODUCED(stage=ci_env, base=origin/main, refined_command=...) — base is origin/main, not the wheel's TORCH_COMMIT_ID the tree is currently detached at |
PASSED |
Return NOT_REPRODUCED(checked_stages=[nightly, source_build, ci_env]) — issue no longer exists; orchestrator reports to user or triage collects reason |
Output
Return one of these to the orchestrator:
REPRODUCED
stage: nightly | source_build | ci_env
base: origin/main | <ci_commit_sha> # base for downstream build. Default origin/main (also for stage=ci_env). ci_commit_sha only when stage=source_build fell back to ci_commit. Stage 3's TORCH_COMMIT_ID wheel-alignment checkout is never returned as base.
refined_command: <single shell-executable string>
refined_command contract. A single shell-executable string that,
run by itself, reliably triggers the failure. A downstream skill (a
fix-verifier, a skip-list per-entry runner, etc.) invokes it directly
(e.g. via bash -c "$refined_command") after applying a candidate fix
to check whether the failure is gone. Consequences:
- Include everything needed to reproduce. Env vars go as inline
prefix (
ZE_AFFINITY_MASK=0 pytest ...), a required cwd goes as acd <dir> &&prefix. - Do NOT include setup steps.
git clone,pip install, wheel-download, etc. that appeared in the input shell block are excluded. The caller has already paid that cost; refined_command should re-trigger the failure, not re-provision the environment. - No
docker runwrapper. Stage 3 runs the reproducer directly in the current shell (the CI job is already inside its container, and the caller of refined_command runs from an equivalent env). If the caller needs container-level isolation, that is its concern, not refined_command's. - Not just the input command. For the pytest form, refined_command
may add
-sv,--timeout <N>, or-xthat Stage 1 used to get a usable failure signal. For the shell-block form, refined_command is the extracted "reproduce step" (usually the last line), not the whole block. - Quoting: use double quotes for inline python. When the reproducer
embeds python code, write it as
python -c "..."(double quotes), notpython -c '...'. Downstream callers wrap the string inbash -c "$refined_command"; single-quoted python payloads compose poorly through that wrapping.
NOT_REPRODUCED
checked_stages: [nightly] | [nightly, source_build, ci_env]
reason: <what was checked and confirmed to pass>
NO_REPRODUCER
reason: no_command | collected_zero
(returned when either no reproducer_command was provided, or pytest
reports `collected 0 items` for the provided command)
CANNOT_VERIFY
stage: nightly | source_build | ci_env
blocker: <what went wrong>
The orchestrator decides the next step based on this output.