failure-classify
Triage failing tests so a real regression does not get lost among
tests that are expected to fail (because they verify that the system
rejects bad input).
When to use
- After every test run that produced at least one failing test.
- Before triaging a CI failure manually.
- In the report-render skill's input pipeline.
Do not use this skill to decide whether a regression is "important
enough to fix". That is a human judgement. The skill only sorts.
Inputs
| Name |
Required |
Default |
Description |
log_path |
yes |
none |
Path to the test runner's output. The skill scans for ✘ lines (or FAIL lines, depending on the language pattern). |
language |
no |
swift |
One of swift, python, go, rust. Picks the line-extraction regex and the classification patterns. |
self_test |
no |
false |
When true, runs the skill's internal unit tests and exits. Use after editing the classification rules. |
Output
| Channel |
Content |
| stdout |
A JSON object with three keys: failures (deduplicated list of failing test names), failures_by_class (counts per class), failures_grouped (test names bucketed by class). |
| exit code |
0 on success regardless of test outcomes. 1 if the log could not be read. 2 on self-test failure. |
Classification taxonomy
Seven classes. Order matters: the first matching pattern wins. A test
name that matches no pattern falls into ASSERTION_FAILURE by default.
| Class |
Meaning |
Trigger keyword (regex fragment) |
EXPECTED_FAILURE |
the test verifies the system rejects something |
Reject, Refuse, ErrorContains, WithInvalid, DataCorrupted, InvalidDuration, LongSessionOnBattery |
IO_BACKEND |
the test exercises a system API (IOKit, libusb, raw sockets) |
language-specific prefix, e.g. powerAssertion* in Swift |
ENVIRONMENT_ERROR |
the test depends on a live system reading |
PowerSourceMonitor, BluetoothState, NetworkLink |
TEST_SCAFFOLD |
the test depends on a missing setup artifact |
reserved; no patterns in v0.1 |
ASSERTION_FAILURE |
real bug |
default bucket |
PRECONDITION_MISSING |
required env var / file is missing |
requireEnv, skipIf markers |
UNKNOWN |
could not classify |
fallback (rare; usually means the runner produced an unexpected line format) |
Algorithm
- Extract failing test names from the log. Swift Testing emits two
✘ lines per failing test (one for the issue, one for the
summary). The skill deduplicates while keeping the first-seen order.
- Classify each name against the patterns, first-match-wins.
- Emit JSON: list, counts, grouped.
Worked example (Swift)
Log:
✔ Test foo() passed after 0.001 seconds.
✘ Test barRejectsZero() failed after 0.001 seconds with 1 issue.
✘ Test bazShouldBehave() failed after 0.001 seconds with 1 issue.
Output:
{
"failures": ["barRejectsZero", "bazShouldBehave"],
"failures_by_class": {
"EXPECTED_FAILURE": 1,
"ASSERTION_FAILURE": 1
},
"failures_grouped": {
"EXPECTED_FAILURE": ["barRejectsZero"],
"ASSERTION_FAILURE": ["bazShouldBehave"]
}
}
The first test is an expected-failure pattern (Rejects). The second
is a real regression candidate.
Anti-patterns
- Treating
EXPECTED_FAILURE as a green light. A failure classified
as EXPECTED_FAILURE still failed. The classification only means the
failure is the test's purpose, not that the system under test
behaves correctly. The test author must read the message.
- Adding patterns that match everything. Patterns are first-match
wins. A pattern like
.* would swallow every test into one bucket.
- Assuming a runner that is not in the language list still works.
The v0.1 patterns cover Swift Testing only. For other runners, the
user must extend the patterns and add a self-test.
Cross-references
- report-render — consumes the JSON
output and embeds it in the Markdown report.
- drill — uses this skill to verify the loop
reacts to a real failure.
- drift-check — the previous step in the
loop.
1---2name: failure-classify-23description: Group failing tests by naming convention so a real assertion failure is not hidden among tests that exist to verify a system rejects something. Use after running the test suite and before reading the report.4---56# failure-classify78Triage failing tests so a real regression does not get lost among9tests that are *expected* to fail (because they verify that the system10rejects bad input).1112## When to use1314- After every test run that produced at least one failing test.15- Before triaging a CI failure manually.16- In the report-render skill's input pipeline.1718Do **not** use this skill to decide whether a regression is "important19enough to fix". That is a human judgement. The skill only sorts.2021## Inputs2223| Name | Required | Default | Description |24| --- | --- | --- | --- |25| `log_path` | yes | none | Path to the test runner's output. The skill scans for `✘` lines (or `FAIL` lines, depending on the language pattern). |26| `language` | no | `swift` | One of `swift`, `python`, `go`, `rust`. Picks the line-extraction regex and the classification patterns. |27| `self_test` | no | `false` | When `true`, runs the skill's internal unit tests and exits. Use after editing the classification rules. |2829## Output3031| Channel | Content |32| --- | --- |33| stdout | A JSON object with three keys: `failures` (deduplicated list of failing test names), `failures_by_class` (counts per class), `failures_grouped` (test names bucketed by class). |34| exit code | `0` on success regardless of test outcomes. `1` if the log could not be read. `2` on self-test failure. |3536## Classification taxonomy3738Seven classes. Order matters: the first matching pattern wins. A test39name that matches no pattern falls into `ASSERTION_FAILURE` by default.4041| Class | Meaning | Trigger keyword (regex fragment) |42| --- | --- | --- |43| `EXPECTED_FAILURE` | the test verifies the system rejects something | `Reject`, `Refuse`, `ErrorContains`, `WithInvalid`, `DataCorrupted`, `InvalidDuration`, `LongSessionOnBattery` |44| `IO_BACKEND` | the test exercises a system API (IOKit, libusb, raw sockets) | language-specific prefix, e.g. `powerAssertion*` in Swift |45| `ENVIRONMENT_ERROR` | the test depends on a live system reading | `PowerSourceMonitor`, `BluetoothState`, `NetworkLink` |46| `TEST_SCAFFOLD` | the test depends on a missing setup artifact | reserved; no patterns in v0.1 |47| `ASSERTION_FAILURE` | real bug | default bucket |48| `PRECONDITION_MISSING` | required env var / file is missing | `requireEnv`, `skipIf` markers |49| `UNKNOWN` | could not classify | fallback (rare; usually means the runner produced an unexpected line format) |5051## Algorithm52531. **Extract** failing test names from the log. Swift Testing emits two54 `✘` lines per failing test (one for the issue, one for the55 summary). The skill deduplicates while keeping the first-seen order.562. **Classify** each name against the patterns, first-match-wins.573. **Emit** JSON: list, counts, grouped.5859## Worked example (Swift)6061Log:62```63✔ Test foo() passed after 0.001 seconds.64✘ Test barRejectsZero() failed after 0.001 seconds with 1 issue.65✘ Test bazShouldBehave() failed after 0.001 seconds with 1 issue.66```6768Output:69```json70{71 "failures": ["barRejectsZero", "bazShouldBehave"],72 "failures_by_class": {73 "EXPECTED_FAILURE": 1,74 "ASSERTION_FAILURE": 175 },76 "failures_grouped": {77 "EXPECTED_FAILURE": ["barRejectsZero"],78 "ASSERTION_FAILURE": ["bazShouldBehave"]79 }80}81```8283The first test is an expected-failure pattern (`Rejects`). The second84is a real regression candidate.8586## Anti-patterns8788- **Treating `EXPECTED_FAILURE` as a green light.** A failure classified89 as `EXPECTED_FAILURE` still failed. The classification only means the90 failure is the test's *purpose*, not that the system under test91 behaves correctly. The test author must read the message.92- **Adding patterns that match everything.** Patterns are first-match93 wins. A pattern like `.*` would swallow every test into one bucket.94- **Assuming a runner that is not in the language list still works.**95 The v0.1 patterns cover Swift Testing only. For other runners, the96 user must extend the patterns and add a self-test.9798## Cross-references99100- [report-render](../report-render/SKILL.md) — consumes the JSON101 output and embeds it in the Markdown report.102- [drill](../drill/SKILL.md) — uses this skill to verify the loop103 reacts to a real failure.104- [drift-check](../drift-check/SKILL.md) — the previous step in the105 loop.
majiayu000