# Ut Check

> Analyze UT (unit test) results for a torch-xpu-ops PR. Use when asked to check test results, analyze CI failures, or evaluate test coverage for a PR. Produces a structured report of new failures, failure relevance, and new test coverage.

- Skill: `intel/ut-check` (Agent Skill)
- Install (CLI): `npx skillmds@latest add intel/ut-check`
- Raw SKILL.md: https://api.skillmd.com/api/skills/intel/ut-check/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: intel (https://skillmd.com/u/intel)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/intel/ut-check

---


# UT Result Check Skill

Analyze unit test results for a torch-xpu-ops PR: identify new failures, assess
whether they relate to the PR changes, and verify new test coverage.

## Quick start

1. Read the UT data JSON at the path given in the request (default
   `/tmp/ut_data.json`). If it is missing or unreadable, say so and stop --
   never reconstruct the data from elsewhere.
2. Analyze it following the workflow below. The data is self-contained; no
   sub-agents or extra CI queries are needed.
3. Emit the report as your final message. The calling workflow posts it as the
   PR comment -- do not post a comment yourself.

## Input

The UT data is provided as a JSON file (typically `/tmp/ut_data.json`) produced
by `.github/scripts/bot_ut_check.py`. The script already de-duplicates failures,
classifies each failure's relevance, summarizes new-test coverage, and computes
a baseline verdict. The JSON contains:

```json
{
  "pr_number": 1234,
  "run_id": 56789,
  "failures": [
    {"category": "op_ut", "class": "a.b.TestFoo", "test": "test_bar",
     "status": "failed", "message": "...", "relevance": "Related"}
  ],
  "changed_files": {
    "operator_source": ["src/ATen/native/xpu/Foo.cpp"],
    "test_files": ["test/xpu/test_foo_xpu.py"],
    "skip_lists": [],
    "other": []
  },
  "new_tests": ["TestFoo::test_bar"],
  "new_tests_summary": {
    "passed": ["TestFoo::test_bar"],
    "failed": [],
    "skipped": [],
    "not_run": []
  },
  "passed_tests_count": 12345,
  "totals": {"test_cases": 0, "passed": 0, "skipped": 0,
             "failures": 0, "errors": 0},
  "verdict": "Safe to merge",
  "verdict_reason": "No new failures detected."
}
```

### Ground Truth for New Failures

The authoritative source for new failures is the `New-UT-Failures-*` artifact
which contains `new_ut_failure_list.csv`. This CSV is produced by the CI
summary job: `ut_result_check.sh` identifies new failures (not in the known
issues list), and the workflow enriches them with error messages.

The `Inductor-XPU-UT-Data-*` artifact bundles a **duplicate** copy of that CSV;
`bot_ut_check.py` reads only the authoritative artifact and de-duplicates by
`(category, class, test)`. If you ever see the same failure listed twice, treat
it as a single failure and note the collection anomaly.

## Analysis Workflow

### Step 1: Read the Data

Read the JSON. Understand the scope: failure count, changed files, new tests.

### Step 2: Review / Refine Failure Relevance

Each failure already carries a deterministic `relevance`
(`Related` / `Possibly related` / `Unrelated`) computed from the changed files.
Trust it as a baseline, but refine using the error `message` and your judgment:

- **Related** -- failure is in a test module that directly tests an operator or
  test file this PR modifies.
- **Possibly related** -- failure is in a related subsystem or a plausible side
  effect of the change.
- **Unrelated** -- failure is in an unrelated module; likely pre-existing flake
  or infrastructure issue (e.g. missing shared library, download error).

If you change a classification, briefly say why.

### Step 3: Evaluate New Test Coverage

Use `new_tests_summary`: `passed`, `failed`, `skipped`, `not_run`. Newly added
tests that FAILED or did NOT RUN are a concern and must be called out.

### Step 4: Produce the Report

Follow the output format below exactly. Apply the truncation rules strictly.

## Guardrails: Do Not Post Low-Quality Comments

- **Never fabricate or guess** counts. Report exactly what the JSON contains.
- If `failures` is empty **and** `new_tests` is empty, produce a concise report
  (New Failures: none; Recommendation) rather than padding with empty tables.
- If the data looks internally inconsistent (e.g. a failure count that seems
  implausible for the diff, or `totals` missing), state the uncertainty plainly
  in the Recommendation instead of asserting a confident verdict.
- Do not invent a "Failure Relevance Analysis" for failures that are all clearly
  unrelated; a one-line grouping is enough.

## Truncation Rules

- **New failures**: If more than 20, show the first 20, then add:
  `... and N more failure(s). See CI logs for the full list.`
- **New/modified tests**: If more than 20, show the first 20, then add:
  `... and N more new test(s). See CI logs for the full list.`
- **Changed files**: Summarize as counts per category.

Always include the total count so the reader knows the full scope.

## Output Format

Every report MUST include: the New Failures section (with the `Related to PR?`
column), a New Test Coverage summary line with counts whenever the PR adds
tests, and a Recommendation with an explicit safe-to-merge verdict.

```markdown
## UT Result Check: PR #<number>

### New Failures
<count> new failure(s) detected (not in known issues). / No new failures detected.

| Test | Category | Status | Related to PR? |
|------|----------|--------|----------------|
| `ClassName::test_name` | category | failed | Related / Possibly related / Unrelated |
...

### Failure Relevance Analysis
Brief explanation, grouping related failures together. Omit if there are no
failures or all are trivially unrelated (a one-line note then suffices).

### New Test Coverage
This PR adds/modifies **<N>** test(s): <p> passed, <f> failed, <s> skipped, <n> not run.

| New/Modified Test | Status |
|-------------------|--------|
| `ClassName::test_name` | PASSED / FAILED / SKIPPED / NOT RUN |
...

### Recommendation
**<Safe to merge | Likely safe to merge | Investigate before merging | Not safe to merge>.**
One-to-two sentence justification grounded in the failures and new-test results.
```

**Omit sections that have no content.** If the PR adds no tests, omit "New Test
Coverage". Always keep the "New Failures" and "Recommendation" sections.

If a calling workflow explicitly requires a skill marker, append this exact
literal final line:
Custom skills applied: ut-check.

Otherwise, keep the reply in the requested report format and do not force an
extra trailing sentence.