Root Cause And Fix

Root-Cause And Fix

Overview

Dev-only loop for the flowai repo. A red test is a claim about the world, not a fact about the product: the harness may have lost the evidence, the checklist may test yesterday's framework, or the scenario may never have supplied what it demands. Measured splits of instrument to product: 7:1 on 2026-08-20, 1:3 on 2026-08-21, 5:5 on 2026-08-22. Decide the layer first, fix second.

Autonomy contract: invoking this skill IS authorization for the whole loop — reading sandboxes, running single scenarios, editing the layer the diagnosis points at, re-measuring, and committing on a non-main branch. Do NOT pause to ask which hypothesis to test or whether a run may be started; the audit trail is the Hypothesis Board plus the evidence quoted in each finding. Hand back only on a hard blocker — a guard fires (system_health, process_watchdog, a pre-commit hook), the environment is missing (auth, Docker, disk), or the second fix attempt for one failure fails — then emit a STOP-ANALYSIS REPORT.

Rules

1. Raw session over any rendering. judge-evidence.md and report.html are the judge's rendering; ground truth is the transcript the CLI wrote for itself plus the sandbox on disk. Renderings also reorder: a report.html verdict block is NOT the run directory of the same index. Absence in a rendering is not absence in the world — every "the agent never X" claim comes from a tool-call count or a session line, never from a summary.
2. Guards are signals. A guard that blocks the run means conditions are wrong. Never reach for --force, --no-verify or an override env var; report the blocker instead.
3. Fix the layer the cause is in. Product defect → the primitive. Instrument defect → scripts/acceptance-tests/lib/. Stale contract → the scenario. Never quiet a red test by editing whichever of the three is easiest to reach.
4. Never flip a checklist into its opposite. Rewriting "must have X" as "must NOT have X" trades one red for another and hides that the requirement was conditional. Scope the item to what it tests and record every dead version.
5. No test-fitting. Do not hint the answer in userQuery, do not script the persona, do not pre-create what the skill should produce. Change a query only because it was malformed, and say so in the file.
6. Correct the record where the wrong claim lives. A comment that encoded a wrong diagnosis costs the next reader a whole investigation — rewrite it in place with what the evidence shows, and keep the retracted version visible.
7. This file is an output of the loop. Rules 4 and 6 apply to it too: a signature that proves wrong gets corrected in place, and what the loop learns gets written back (Phase 6). A lesson left in chat is lost.

Phase 0 — Instrument or product

Run this BEFORE anything else. Each signature below was paid for once.

• Exit 1, empty trace, ~10-20 s. OAuth session expired, Invalid API key, Usage credits required: source .env before the sweep. detectAuthFailure throws only at zero tool calls — a scenario driving another IDE's CLI surfaces THAT child's error, a true observation.
• Exit 124 with tool calls present. A global timeout, a warning rather than a blocker when the trace is non-empty. Ask whether the work legitimately outlives the cap — deep-research does.
• The judge says something is missing or wrong and the disk says otherwise. It only ever saw the evidence blob, so ask what that blob contained. Five known holes, all fixed, all worth re-checking: a file over 30 KB elided mid-file (renderFileForEvidence); evidence carrying only git diff <init>..HEAD, which hid the product of a workflow that stops before committing (adapt-skills-basic); the TRACE cut head+tail, deleting the middle of the conversation (2026-08-28: 5 of 10 turns survived); a deliverable whose extension is outside TEXT_EXTENSIONS in evidence.ts, which never reaches the judge at all (.html on 2026-09-06 — "HTML contents are not provided", five items red, file on disk); and a [codex-agents] NEW_TASK with an empty Payload:, which is codex ≥ 0.145 ENCRYPTING the prompt in the rollout, not an empty dispatch. Count turns (grep -c '^\[turn' judge-evidence.md) before believing "the agent never did X".
• The cache file is not the run. acceptance-tests/cache/<pack>/<id>/<ide>.json holds the last RECORDED verdict, often an older green one, and contradicts the run without saying so. Take per-item verdicts from the run, never the cache.
• Adjacent-negative fails, correct neighbour lives in another pack. The runner mounts core plus the scenario's pack, so there was nothing to defer to. Set extraPacks (FR-ACCEPT.TRIGGER).
• Checklist demands an artefact the primitive's own text forbids. Read the SKILL.md first: init forbids wrapper scripts when the project's runner suffices, while its checklist demanded scripts/check.ts.
• The red scenario is untracked. git ls-files does not list it — another session's deliberate RED, not your regression. Never fix it.
• A verdict that changed with no tree change. Load noise — a real defect fails the same way three runs running. Then ask what the load EXPOSED: re-measuring hides a deterministic cause as readily as it clears a false one. The watchdog's rss test was dismissed as noise for months, while every green re-run was a poll landing inside the second before the OS compressed its buffer away.
• The turn ends while a dispatched subagent is still running. The ACP loop runs only while the emulator answers, so the result is never collected (maintenance-basic died at 228 s where siblings ran 950 s). Cure it in the primitive: "wait for all subagents" must say waiting means COLLECTING, and that nothing is presented while one is outstanding.
• The fixture is not what the scenario thinks it is. git show <init-sha>:<file> in the sandbox is the exact input. setup() runs AFTER that commit, so what it plants is an UNCOMMITTED change in the agent's own diff — the judge read one as a line the agent had WRITTEN and failed the item asking whether it had READ it; planted pre-existing state must be committed. The fixture must also pass every deterministic checker or the agent stops on THAT and never reaches the planted defect (three of ten scenarios on 2026-08-22).
• Exit 144, no verdict, a [fork-loop guard] line. The guard aimed at the BENCH (fixed by adoptablePgid), or at a legitimate shell pipeline (fixed by raising maxDescendants to 16 on 2026-08-25).
• Zero tool calls. At 0.0 s system_health refused the spawn and quoted its reason INSIDE the checklist ERROR (load avg N/CPU > 4/CPU), reading as a finding about the product; families dispatching subagents (plan-*, ship-task-*) trip it at -p 2 and above, so re-measure at -p 1. At full duration and exit 0 the agent stopped instead: review_ready means it produced the artefact unaided, blocked means the query never supplied what it asked for. With a userPersona, check interactive — UserEmulator exists only when that flag is true, so otherwise the persona is dead text.
• A mocks entry does not stand in for the whole pipeline. writeMockBin shadows the binary on PATH, so the canned text is the next pipe stage's INPUT. A mock holding already-parsed output passes only while the agent skips the pipe.
• Every judge call fails, or a probe hangs into aborted_streaming. Read the agent's own text before blaming the CLI: API Error: 529 Overloaded sits there in plain words. grep -l "API Error: 5" over a sweep bounds it.
• A scenario at 2/3 passes the threshold and can still hold a real defect. Read the failing run. A fix that MOVES a red to 2/3 has the same shape: it cured one real cause and uncovered the next. Attempts on two different causes are not two attempts at one, so STOP-ANALYSIS does not fire — but say which cause each attempt addressed, or the count is unauditable.
• The rule was in the file and still did not fire. First prove the text reached the agent — a green sibling sharing the artefact does it in one line. Then read the session for the shape. Never mentions it → never bound; bind it AHEAD of the decision, since an agent holding a solution reads a rule for exemptions. Quotes and overrides it → say what compliance PRODUCES, and grep for a neighbouring rule claiming the case, since the weaker of two wins. Reclassifies the case → your own carve-out is the escape; scope it to what the user NAMED. Forbid the act, not the harm: a rule stated as a consequence dies by denying the consequence.

Print the verdict — INSTRUMENT or PRODUCT — with the evidence line that decided it, then continue.

Phase 1 — Evidence

Read in this order and stop as soon as the cause is unambiguous.

1. The raw session. <run>/<scenario>/run-N/bench-home is a SYMLINK into $TMPDIR/flowai-bench/run-N-<hash>/; readlink it, then find the .claude/projects/<slug>/<uuid>.jsonl inside. Tool histogram (add and .name=="Skill" piped to .input for skill calls; subagent transcripts sit under <uuid>/subagents/): jq -r 'select(.message.content|type=="array") | .message.content[] | select(.type=="tool_use") | .name' <file> | sort | uniq -c | sort -rn
2. The sandbox on disk (readlink <run>/<scenario>/run-N/sandbox) — what the agent wrote settles claims no transcript reading can.
3. The failed agent itself, when the diagnosis is about wording. Resume in place (cd <sandbox> && HOME=<bench-home> claude -p --resume <uuid> "<question>") and ask neutrally which phrase left room — three runs named the same defective sentence in one round where three rounds of guessing had not. The transcript shows what it did, the interview the words it justified it by.
4. If the run dir was pruned, the transcript and per-item JSON survive in report.html — parse with python, never grep raw HTML, and never Read a .jsonl (104 000 tokens for 44 lines).
5. The scenario file: what it asserts, and whether the query supplies what it demands. Then the primitive's own text — the atom under framework/atoms/, never the generated SKILL.md.

Phase 2 — Hypotheses

Propose 3–7 candidate causes with probabilities summing to ~100, one line of reasoning each, and print the Hypothesis Board. For the highest-probability untested one, design an experiment with a discrete outcome — state before running it what success and failure each prove. Execute, record, re-weight, reprint. With several reds in hand, check each isolates a different hypothesis. Terminate at ~80 %, after three experiments that move nothing, or 5 iterations.

Diagnostic edits are reverted from cp backups — never with git checkout -- or git restore, which return the index rather than what you wrote.

Phase 3 — Fix

• Product: edit framework/atoms/<name>.md (or the skill/agent file), then deno run -A scripts/generate-skill-composites.ts --write; the nine generated SKILL.md files are gitignored build artefacts (--list-targets names them).
• Instrument: edit scripts/acceptance-tests/lib/. Extract the decision into a pure exported function and unit-test it in a file deno task check runs, NOT runner_test.ts, which task-check.ts ignores.
• Contract: edit the scenario or fixture, saying in the file what the old version held and why it was wrong.

When the layer is a RULE that did not fire, two shapes decide whether the repair holds. Prefer precondition grammar to prohibition — "X is a precondition on Y, not a step you may or may not enter" binds where "you are NOT done until X" does not. And elaborate a shared paragraph on ALL its triggers or none: two sentences about a missing VALUE made the same paragraph stop binding for a missing SCRIPT, and a green scenario fell to 1/3 for a case I never touched.

Every fix carries, in the file it touches, the measurement that justified it: date, runs, what the sessions showed. Then grep the file for the NEW text — the script's exit code is not evidence the edit landed. One replace was computed and never applied, and a whole measurement round paid for it. A framework rule may not name a project's doc paths either (FR-UNIVERSAL.DOC-SCHEMA) — that fails check-skills.

Phase 4 — Verify

• Single scenario, cache bypassed, in the background, .env sourced in the same shell (they do not persist): deno task acceptance-tests -f <id> -n 3. -f takes ONE substring, last wins; -p sets concurrency; the lock forbids concurrent runs. A foreground run past the tool's cap is killed mid-flight.
• Host preflight: load, free swap, orphaned runners (ps -Ao pid,etime,command | grep -E "runtests.py|benchmark.ts run"). Under pressure system_health aborts sessions and every result is noise; ten of eleven such failures retried green.
• The verdict is the run log's --- PASS RATES --- block, not report.html. The threshold is 2/3, so [PASS] covers a run that failed a critical item, and a section whose verdict JSON you cannot scrape is not a red — read the log.
• An instrument fix is verified by its unit test AND by the scenario it misread; a product fix, against the raw sessions and the sandbox. The number has to move for the reason claimed. Re-measure one GREEN sibling of every primitive you edited — that guard caught a scenario dropping 3/3 to 1/3.
• Name what the green number covers, and build the task to cover it. A rule phrased "if you see X" binds at every stage while the suite holds one, so "3/3" reads general when it is not. An A/B asking whether a primitive still earns its place must exercise EVERY hazard it addresses: a one-hazard task the bare model already handles returned 3/3 against 3/3 on 2026-08-30 and read as "the primitive changes nothing"; a four-hazard task split the arms 0/3 to 3/3.
• deno task check before every commit. Its verdict is the final N passed | M failed line; the three === FAIL deno eval Deno.exit(...) lines are intentional fixtures.

Phase 5 — Record and commit

• Update the docs the change maps to (AGENTS.md Documentation Map): an instrument change in SDS §3.4, a trigger lesson in FR-ACCEPT.TRIGGER. When a doc calls the behaviour you are changing deliberate, say so and ask.
• Check git diff --cached --stat in a SEPARATE tool call, then commit by explicit paths. The tree is shared with other sessions.
• The commit message states what was wrong, what the evidence was, and what remains red. A fix whose measurement did not move says so.

Phase 6 — Amend this skill

This file is the only artefact that carries a lesson into the next session. Decide at the end of every run whether it should change and say which way — a silent skip is indistinguishable from a forgotten step.

Earns an edit: a cause Phase 0 would NOT have caught, that can recur (one bullet, dated); a rule you broke that cost real work; a signature here that proved wrong — correct it in place, keeping the retracted claim visible; a command, path or flag that changed under you.

Does not earn an edit: a one-off in one environment; a restatement of a rule already here; a finding about the PRODUCT rather than about diagnosing it; a run that went well.

Budget: 270 lines, raised five times (240 → 270 across 2026-08-24/25) and never again. The ratchet is the failure the budget exists to prevent, so the next addition DISPLACES a bullet. Compress first, never as a fallback: two bullets on one failure shape are one bullet, and on 2026-08-30 that merge paid for three new signatures at no net cost. Commit the amendment with the work that produced it and say what was learned, not that the skill was updated.

Verification