# Verify Completion Signal

> Use after any long-running subprocess, eval driver, CI gate, or job pipeline run reports apparent success (exit 0 / phase=DONE / "all tests passed") — before accepting the result. Catches the silent-success trap: a process can exit 0 without completing its intended work, and inference from "no error" to "success" is wrong. Require an explicit positive completion artifact (an output file, pass-count line, verdict.json with a status field, non-empty results directory).

- Skill: `oimiragieo/verify-completion-signal` (Agent Skill)
- Install (CLI): `npx skillmds@latest add oimiragieo/verify-completion-signal`
- Raw SKILL.md: https://api.skillmd.com/api/skills/oimiragieo/verify-completion-signal/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: oimiragieo (https://skillmd.com/u/oimiragieo)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/oimiragieo/verify-completion-signal

---


# Verify a Completion Signal

## Overview

Exit 0 means "the process finished without crashing." It does not mean "the outcome you wanted exists in the world." The gap between those two statements is where silent failures live — and they look identical to successes until you check the artifact directly.

**Core principle:** require a POSITIVE completion signal. An absence of error is not a positive signal. A positive signal is a concrete artifact that can only exist if the work was done: a file with non-zero size, a `X passed` line in stdout, a `verdict.json` with a populated `status` field, a non-empty results directory.

This is a cross-cutting reflex. It applies to any tool, subprocess, CI gate, or eval driver that can exit cleanly without completing its work.

## When to use

- A long-running job pipeline reaches `phase=DONE` or `exit=0` — before accepting the result, check the output directory contains actual result files.
- A pytest product-gate appears to exit 0 with no `X passed` line in stdout (faulthandler/pluggy crash path: the interpreter exits before the test runner prints its summary).
- A driver prints `=== DONE ===` but the results directory is empty or contains only the log.
- Any subprocess piped through `tee`, `||  true`, or a shell wrapper — the last command's exit code shadows the real one.
- An agent or eval loop self-reports "complete" or "success" with no artifact cited.

NOT for: validating that the outcome is *correct* once the artifact exists — use the task's own metric after the completion signal is confirmed.

## The method

**Step 1 — Name the artifact.** Before the run starts, state what must exist on disk when it succeeds. Not "the script ran." The real thing: `output_dir/master.out` + at least one `.json` in `output_dir/`, a `X passed` line in captured stdout, a non-empty `results/<run_id>/` directory.

**Step 2 — After apparent success, check the artifact independently.** Do not re-read the tool's own return value or log line. Go to the filesystem / stdout buffer directly:
- `ls -lh output_dir/` — does it contain files with non-zero size?
- Grep stdout for `passed` (not just exit code): `echo "$OUTPUT" | grep -E '[0-9]+ passed'`
- Read `verdict.json` and confirm `status` is populated, not a placeholder like `STEP1_INCOMPLETE`.

**Step 3 — Treat a missing artifact as a failure, not as "probably fine."** If the artifact is absent, the run did not complete — even if exit=0, even if phase=DONE. A DONE marker with an empty output dir is a bug, not a success.

**Step 4 — Identify the silent-success class and fix it upstream.** Three common classes:

| Class | Symptom | Root cause | Fix |
|---|---|---|---|
| DONE-with-empty-output | `phase=DONE`, `exit=0`, output dir has no result files | the pipeline's DONE gate does not verify `n_files > 0` in the output dir | Until fixed: `ls output_dir/` manually after every run; count non-log files |
| Gate-exit-0-but-crashed | `pytest` exits 0, stdout has no `X passed` line | native faulthandler / pluggy crash (`_pytest.faulthandler` path) exits the interpreter before the test summary prints | Require the `X passed` pattern in captured stdout; a zero-line output is not a pass |
| Pipe-swallowed exit | `cmd \| tee log.txt` exits 0 regardless of `cmd`'s code | bash returns the exit of `tee`, which always succeeds | `set -o pipefail` in every shell driver; capture `${PIPESTATUS[0]}` explicitly |

## Common mistakes

**"Exit 0 = success."** Exit 0 = "the process terminated without an unhandled signal or explicit failure exit." A script that finds nothing to do, a process that crashes via `_exit()` before cleanup, and a subprocess wrapped in `|| true` all exit 0. None of them completed the work.

**"The DONE marker was printed."** The DONE marker in `master.out` (`=== DONE === run=R exit=N`) means the driver reached its terminal block and printed. It does not mean the result-transfer step succeeded or that the output directory has result files. A verdict-classifier function that checks the *content* of transferred results — but only if files exist — can silently return `None` on an empty output dir, masking the gap by falling through to a generic DONE outcome.

**"All tests passed" (from the agent's summary, not stdout).** An LLM summarizer reading test output can report success from a partial log — if the stdout was truncated or the crash happened after the test collector ran but before the reporter printed. The `X passed` line must appear in the raw captured stdout, not in a summary inferred from it.

**"The transfer verified it."** A results-transfer verification step can check only the files listed in an `expected_files` list. If that list only names a log file and the actual results JSON was never written, the check verifies the log and declares success. The check is only as complete as the list you pass in.

## Field grounding

- arXiv:2509.14347 (Illusion of Success, 2025): 11% of industrial CI jobs are rerun; "silent failures" — exit 0 but the intended artifact is missing — are the top-3 cause: artifact operation errors 28%, caching errors 23%, ignored non-zero exit codes 18%.
- The `| tee` class is the canonical CI trap (three weeks of broken builds shipped green: bash returns `tee`'s exit, not the compiler's; the fix is `set -o pipefail`).
- pytest faulthandler path: when a C extension or pluggy hook crashes the interpreter before the test-session end hook runs, `_pytest.faulthandler` may write a traceback to stderr but the Python process exits via `_exit(status)` — which skips atexit, `sys.exit`, and the test reporter. The exit code may be 0 if the crash happened in teardown rather than in a test itself.

## Receipts

- A long-running job runner reached `phase=DONE exit=0` with zero result files transferred back to the output directory. The verdict-classifier silently returned `None` on an empty output dir, so the final outcome fell through to a generic DONE. No `n_files > 0` guard existed yet.
- A full product-gate test suite faulted via a native pluggy crash — exited 0 with no `X passed` line in stdout. The gate appeared green; it delivered no result.
- Both were caught by the same reflex: "the run says DONE / green — where is the artifact?"

Companion skill: `verify-the-claim-not-the-label`.

## The positive-label corollary (see `verify-the-claim-not-the-label`)

This skill verifies COMPLETION artifacts after a run (exit codes, DONE markers, output files). The same reflex extends to any CACHED or ASSERTED positive-state LABEL — 'this is resolved', 'this is passing', 'the cache is ready', 'the cause is X', including labels YOU just applied. That generalization lives in the canonical skill **`verify-the-claim-not-the-label`**. Rule: before stating a bug is 'resolved' / a test is 'passing' / a mechanism is 'X', confirm the positive artifact (the 'N passed' line, the per-sample distribution, the file size > 0) — not the label. A label is a hypothesis.