Distributed Triage

Distributed Issue Triage Sub-Skill

This sub-skill picks up where the PT-level triage bot leaves off. It processes issues that already have the oncall: distributed label and performs second-level triage: routing to a distributed sub-oncall, classifying by module, and marking triaged.

Contents

• MCP Tools Available
• Reference Files
• Comment Deduplication
• Distributed Triage Steps
  • Step 0: Already Triaged by Human
  • Step 1: Is This Actually a Distributed Issue?
  • Step 2: Route to Distributed Sub-Oncall
  • Step 3: Classify Module
  • Step 4: Type Labels
  • Step 5: High Priority — REQUIRES HUMAN REVIEW
  • Step 6: Missing Reproduction
• Constraints

Distributed labels reference: See distributed-labels.json for the labels this skill is allowed to apply. ONLY apply labels from this file.

Distributed triage rubric: See distributed-rubric.md for detailed routing guidance, module classification signals, and confidence calibration.

Response templates: See templates.json for distributed-specific comment templates.

MCP Tools Available

Use these GitHub MCP tools for triage:

Comment Deduplication

Before adding any issue comment:

1. Read the existing comments with mcp__github__get_issue_comments.
2. Check whether the triage bot has already posted the same template or a substantially equivalent request/explanation.
3. If a duplicate exists, do not add another comment. Continue with any non-comment actions that are still needed, such as labels.

Treat a comment as duplicate even if the wording differs slightly or an older template version was used. For distributed triage, this includes an existing distributed reproduction request or an existing "not distributed" notice.

Distributed Triage Steps

0) Already Triaged by Human?

A human has fully classified the issue only when it has BOTH:

1. Any module: label listed in distributed-labels.json, AND
2. One of the sub-oncall labels: oncall: distributed parallelisms, oncall: distributed infra, or oncall: distributed checkpointing.

If both are present:

• Add bot-triaged + triaged labels (the human classification is complete and confident)
• STOP — a human already classified this issue.

If only one is present (a module label without a sub-oncall, or a sub-oncall without a module label), triage is incomplete — proceed to Step 1. The PT-level triage bot can apply distributed module labels alongside oncall: distributed, but it does not pick the sub-oncall; that is your job.

This step alone should clear a large portion of the backlog.

1) Is This Actually a Distributed Issue?

Read the issue title, description, and comments. Determine whether the issue is actually related to distributed training.

Signs it is NOT a distributed issue:

• Single-GPU issue with no distributed code (e.g., torch.nn on one GPU, CUDA OOM on one device)
• Build/packaging issue (e.g., undefined symbol: ncclAlltoAll at import torch with no distributed code)
• Pure torch.compile issue with no distributed component
• Issue about a domain library (vision, text, audio) that happens to mention "distributed"

If NOT a distributed issue:

1. Add triage review + bot-triaged labels
2. Post a comment using the not_distributed template from templates.json, unless an equivalent "not distributed" comment already exists
3. Do NOT remove oncall: distributed — let the human oncall re-route
4. STOP

2) Route to Distributed Sub-Oncall

Each issue carries exactly ONE sub-oncall label. If the issue already has one of the three sub-oncall labels (oncall: distributed parallelisms, oncall: distributed infra, or oncall: distributed checkpointing), keep it as-is — do NOT add a second sub-oncall, even if your own classification would have picked a different one. Use the existing sub-oncall to decide the next step (continue to Step 3 if it's oncall: distributed parallelisms; otherwise add bot-triaged and STOP per the rules below).

If no sub-oncall is present, apply exactly one based on the routing rules in distributed-rubric.md:

Use the routing decision tree and edge cases in distributed-rubric.md Section 1 to determine the correct sub-oncall.

After routing to oncall: distributed infra or oncall: distributed checkpointing:

• Add bot-triaged (the sub-oncall routing is a confident, complete outcome)
• STOP — the sub-oncall team owns further triage

After routing to oncall: distributed parallelisms:

• Continue to Step 3 for module classification

3) Classify Module

From the issue description, comments, code snippets, and stack traces, classify into one or more distributed modules. Consult the module classification signals in distributed-rubric.md.

Confidence-based actions:

Rules:

• You can apply multiple module labels when the issue spans modules (e.g., module: fsdp + module: dtensor for FSDP2 issues that hit DTensor bugs).
• When an issue has oncall: pt2 already applied, do NOT remove it. Add distributed module labels alongside it.
• When the module is unclear, add triage review + bot-triaged — do NOT guess a module label.

4) Type Labels

If the issue is not a bug report, add the appropriate type label:

• feature — wholly new functionality that does not exist today in any form
• enhancement — improvement to something that already works (e.g., performance optimization, better error messages, adding a native backend for an op that already runs via fallback)

Most distributed issues are bug reports — do not add a type label for bugs. If the issue says the operation "currently works" or "falls back to" a slower path, that is enhancement, not feature. If the enhancement is about performance, also add module: performance.

5) High Priority — REQUIRES HUMAN REVIEW

CRITICAL: If you believe an issue is high priority, you MUST:

1. Add triage review label and do NOT add bot-triaged

Do NOT directly add high priority without human confirmation.

High priority criteria for distributed issues:

• Crash / segfault / illegal memory access in distributed code
• Silent correctness issue (wrong results from collectives, incorrect gradient sync)
• Regression from a prior version (e.g., FSDP worked in 2.x, broken in 2.y)
• Hang affecting multi-node training (NCCL timeout, deadlock in collectives)
• Data corruption during distributed checkpointing
• Internal assert failure in c10d or process group code
• Many users affected or core distributed component impacted

6) Missing Reproduction

If the issue lacks a minimal reproduction script:

1. Add needs reproduction + bot-triaged labels
2. Post a comment using the needs_distributed_reproduction template from templates.json, unless an equivalent distributed reproduction request already exists

Do NOT request reproduction when:

• The issue already has a code snippet, script, or steps that someone could follow to reproduce
• The issue is a feature request (no repro needed)
• A multi-node script is provided (that counts as reproduction even if you can't run it locally)

Constraints

DO NOT:

• Close issues (only the PT-level bot or humans close issues)
• Remove existing labels — only add labels
• Remove oncall: distributed — it stays even if the issue is mislabeled
• Remove oncall: pt2 — if already present, keep it
• Remove bot-triaged or triaged — they are applied by the parent skill and must stay
• Add triaged when you are NOT confident in the classification — i.e. any time the action also applies triage review or needs reproduction, or in the §5 high-priority flow
• Add labels not in distributed-labels.json
• Add comments to issues except when using the templates in Step 1 (mislabel) or Step 6 (reproduction)
• Add a comment when the bot has already posted the same template or a substantially equivalent message on the issue
• Assign issues to users
• Add high priority directly — use triage review and let humans decide

DO:

• Be conservative — when in doubt, add triage review for human attention
• Add bot-triaged whenever the bot has processed the issue, regardless of confidence. Pair with triage review for LOW-confidence or uncertain cases so the cron sweep won't re-pick it. (Exception: §5 high-priority flow intentionally omits bot-triaged.)
• Add triaged ONLY when you reach a confident, complete classification: a human already classified it (Step 0), a confident sub-oncall routing (Step 2), or a HIGH/MEDIUM-confidence module classification (Step 3).
• Always add a sub-oncall label (Step 2) before module labels (Step 3)
• Read the full issue including comments before classifying
• Read existing comments before every comment action and skip duplicate bot messages
• Check the rubric's "Common Mislabel Traps" section before finalizing