Root Cause And Fix

Root-Cause And Fix

Overview

Dev-only loop for the flowai repo. A red test is a claim about the world, not a fact about the product: the harness may have lost the evidence, the checklist may test yesterday's framework, or the scenario may never have supplied what it demands. Measured splits of instrument to product: 7:1 on 2026-08-20, 1:3 on 2026-08-21, 5:5 on 2026-08-22. Decide the layer first, fix second.

Autonomy contract: invoking this skill IS authorization for the whole loop — reading sandboxes, running single scenarios, editing the layer the diagnosis points at, re-measuring, and committing on a non-main branch. Do NOT pause to ask which hypothesis to test or whether a run may be started; the audit trail is the Hypothesis Board plus the evidence quoted in each finding. Hand back only on a hard blocker — a guard fires (system_health, process_watchdog, a pre-commit hook), the environment is missing (auth, Docker, disk), or the second fix attempt for one failure fails — then emit a STOP-ANALYSIS REPORT.

Rules

1. Raw session over any rendering. judge-evidence.md and report.html are the judge's rendering; ground truth is the transcript the CLI wrote for itself plus the sandbox on disk. Renderings also reorder: a report.html verdict block is NOT the run directory of the same index. Absence in a rendering is not absence in the world — every "the agent never X" claim comes from a tool-call count or a session line, never from a summary.
2. Guards are signals. A guard that blocks the run means conditions are wrong. Never reach for --force, --no-verify or an override env var; report the blocker instead.
3. Fix the layer the cause is in. Product defect → the primitive. Instrument defect → scripts/acceptance-tests/lib/. Stale contract → the scenario. Never quiet a red test by editing whichever of the three is easiest to reach.
4. Never flip a checklist into its opposite. Rewriting "must have X" as "must NOT have X" trades one red for another and hides that the requirement was conditional. Scope the item to what it tests and record every dead version.
5. No test-fitting. Do not hint the answer in userQuery, do not script the persona, do not pre-create what the skill should produce. Change a query only because it was malformed, and say so in the file.
6. Correct the record where the wrong claim lives. A comment that encoded a wrong diagnosis costs the next reader a whole investigation — rewrite it in place with what the evidence shows, and keep the retracted version visible.
7. Distinguish variance from regression. A real defect fails the same way three runs in a row; one sweep disagreeing with the next on an unchanged tree is load noise. Re-measure — but name the mechanism before settling on noise, because a flake with a mechanism is a defect that load merely uncovers.
8. This file is an output of the loop. Rules 4 and 6 apply to it too: a signature that proves wrong gets corrected in place, and what the loop learns gets written back (Phase 6). A lesson left in chat is lost.

Phase 0 — Instrument or product

Run this BEFORE anything else. Each signature below was paid for once.

• Exit 1, empty trace, ~10-20 s. OAuth session expired, Invalid API key, Usage credits required: source .env before the sweep. detectAuthFailure throws only at zero tool calls — a scenario driving another IDE's CLI surfaces THAT child's auth error, a true observation.
• Exit 124 with tool calls present. A global timeout, and since 2026-08-22 a warning rather than a blocker when the trace is non-empty. Ask whether the work legitimately outlives the cap — deep-research does.
• Judge reports a section "not present" in a large file. Elided mid-file above 30 KB (renderFileForEvidence); check disk first.
• Zero tool calls, exit 0. Ambiguous. Read the turn status: review_ready means the model produced the artefact unaided; blocked means it stopped for material the query never supplied — a malformed scenario, not a routing miss.
• Adjacent-negative fails, correct neighbour lives in another pack. The runner mounts core plus the scenario's pack, so there was nothing to defer to. Set extraPacks (FR-ACCEPT.TRIGGER).
• Checklist demands an artefact the primitive's own text forbids. Read the SKILL.md first: init forbids wrapper scripts when the project's runner suffices, while its checklist demanded scripts/check.ts.
• The red scenario is untracked. git ls-files does not list it: another session's deliberate RED in a shared tree, not your regression. Never fix it.
• A verdict that changed with no tree change. Load noise — then ask what the load EXPOSED, because re-measuring hides a deterministic cause as readily as it clears a false one. On 2026-08-24 the watchdog's rss test had been dismissed as noise for months; the buffer it measured was compressed away by the OS in under a second, and every green re-run was a poll landing inside that window.
• The turn ends while a dispatched subagent is still running. The ACP loop runs only while the emulator answers, so the result is never collected and the final summary never happens (maintenance-basic died at 228 s where siblings ran 950 s; two more primitives did the same on 2026-08-23). The cure is in the primitive: "wait for all subagents" has to say that waiting means COLLECTING, and that nothing is presented while one is outstanding.
• The scenario asserts something the sandbox never contained. The runner commits the fixture, so git show <init-sha>:<file> is the exact input — read it before believing "the agent missed X". setup() runs AFTER that commit, so whatever it plants is an UNCOMMITTED change landing in the agent's own diff: on 2026-08-23 the judge read an injected hierarchy as a line the agent had WRITTEN and failed the item asking whether it had READ it. A setup() planting pre-existing state must commit it. A fixture can also speak a retired dialect its mod.ts says it left, so run any deterministic checker over it, green scenarios included: audit-clean passed while audit.ts found 8 issues on the fixture its checklist calls clean.
• The fixture breaks a SECOND rule behind the first. It must itself pass deno fmt --check, deno lint, deno test and ship the Benchmark Fixture deno.json. Three of ten scenarios needed two rounds on 2026-08-22 — the agent stopped on THAT and never reached the planted defect.
• Exit 144, no verdict, a [fork-loop guard] line naming one pgid for two rootPids. The guard aimed at the BENCH: under load setpgrp_exec.py reaches setsid() after the first tick, so the watchdog cached the bench's own group. Only at -p >1; fixed by adoptablePgid.
• The agent asked a question and stopped, and the scenario has a persona. UserEmulator is built only when interactive is true (runner.ts), so a userPersona without that flag is dead text. Such a scenario is green only while the skill skips asking; tighten the skill and it goes red for free.
• A mocks entry does not stand in for the whole pipeline. writeMockBin shadows the binary on PATH, so the canned text is the next pipe stage's INPUT. A mock holding already-parsed output passes only while the agent skips the documented pipe; the comment above it may still describe the retired hook.
• Every judge call fails, or a probe hangs into aborted_streaming. Read the agent's own text before blaming the CLI or its flags: API Error: 529 Overloaded sits there in plain words, and on 2026-08-24 it cost half an hour of bisecting --json-schema. grep -l "API Error: 5" over a sweep bounds it.
• A scenario at 2/3 passes the threshold and can still hold a real defect. Read the failing run. Two did on 2026-08-22, both product defects.
• The rule was in the file and still did not fire. First prove the text reached the agent: an unrelated instruction from the same file obeyed in the same session. Then read the session for the shape. Never mentions it → never bound; add a binding moment AHEAD of the decision, since an agent holding a solution reads the rule for exemptions. Quotes and overrides it → say what compliance PRODUCES, and grep for a neighbouring rule claiming the case, since the weaker of two wins. Obeys it and reclassifies the case → your own carve-out is the escape; scope it to what the user NAMED. Forbid the act, not the harm: a rule stated as a consequence dies by denying the consequence, and a disclaimer above a forbidden output does not make it a different output.

Print the verdict — INSTRUMENT or PRODUCT — with the evidence line that decided it, then continue.

Phase 1 — Evidence

Read in this order and stop as soon as the cause is unambiguous.

1. The raw session. <run>/<scenario>/run-N/bench-home is a SYMLINK into $TMPDIR/flowai-bench/run-N-<hash>/; readlink it, then find the .claude/projects/<slug>/<uuid>.jsonl inside. Tool histogram (add and .name=="Skill" piped to .input for skill calls; subagent transcripts sit under <uuid>/subagents/): jq -r 'select(.message.content|type=="array") | .message.content[] | select(.type=="tool_use") | .name' <file> | sort | uniq -c | sort -rn
2. The sandbox on disk (readlink <run>/<scenario>/run-N/sandbox) — what the agent actually wrote settles claims no transcript reading can.
3. The failed agent itself, when the diagnosis is about wording. Resume in place (cd <sandbox> && HOME=<bench-home> claude -p --resume <uuid> "<question>") and ask neutrally which phrase left room. On 2026-08-21 three runs named the same defective sentence in one round, after three rounds of guessing had not: the transcript shows what it did, the interview the words it justified it by.
4. If the run dir was pruned, the transcript and the judge's per-item JSON survive in report.html — parse with python, never grep the raw HTML, and never Read a .jsonl (104 000 tokens for 44 lines).
5. The scenario file: what it asserts, and whether the query supplies what it demands. Then the primitive's own text — the atom under framework/atoms/, never the generated SKILL.md.

Phase 2 — Hypotheses

Propose 3–7 candidate causes with probabilities summing to ~100, one line of reasoning each, and print the Hypothesis Board. For the highest-probability untested one, design an experiment with a discrete outcome — state before running it what success and failure each prove. Execute, record, re-weight, reprint. With several reds in hand, check each isolates a different hypothesis. Terminate at ~80 %, after three experiments that move nothing, or 5 iterations.

Diagnostic edits are reverted from cp backups — never with git checkout -- or git restore, which return the index rather than what you wrote.

Phase 3 — Fix

• Product: edit framework/atoms/<name>.md (or the skill/agent file), then deno run -A scripts/generate-skill-composites.ts --write; the nine generated SKILL.md files are gitignored build artefacts (--list-targets names them).
• Instrument: edit scripts/acceptance-tests/lib/. Extract the decision into a pure exported function and unit-test it in a file deno task check runs, NOT runner_test.ts, which task-check.ts ignores.
• Contract: edit the scenario or its fixture, saying in the file what the old version held and why it was wrong.

Every fix carries, in the file it touches, the measurement that justified it: date, runs, what the sessions showed. Then grep the file for the NEW text — the script's exit code is not evidence the edit landed. On 2026-08-24 one replace was computed and never applied, so the file carried a comment describing a change that was not there and a whole measurement round paid for it. A framework rule may not name a project's doc paths either (FR-UNIVERSAL.DOC-SCHEMA): writing "do not read the default paths" while naming them fails check-skills.

Phase 4 — Verify

• Single scenario, cache bypassed, in the background, .env sourced in the same shell (they do not persist): deno task acceptance-tests -f <id> -n 3. -f takes ONE substring, last wins; -p sets concurrency; the lock forbids concurrent runs. A foreground run past the tool's cap is killed mid-flight.
• Host preflight: load, free swap, orphaned runners (ps -Ao pid,etime,command | grep -E "runtests.py|benchmark.ts run"). Under memory pressure system_health aborts sessions and every result is noise; re-measuring a whole aborted column is cheap, and on 2026-08-24 ten of eleven such failures were green on retry.
• An instrument fix is verified by its unit test AND by the scenario it misread; a product fix, against the raw sessions and the sandbox. The number has to move for the reason claimed. Re-measure one GREEN sibling of every primitive you edited — that guard caught a scenario dropping 3/3 to 1/3 on 2026-08-24.
• Name what the green number covers: a rule phrased "if you see X" binds at every stage while the suite holds one, so "3/3" reads as general when it is not.
• deno task check before every commit. Its verdict is the final N passed | M failed line; the three === FAIL deno eval Deno.exit(...) lines are intentional fixtures.

Phase 5 — Record and commit

• Update the docs the change maps to (AGENTS.md Documentation Map): an instrument change in SDS §3.4, a trigger lesson in FR-ACCEPT.TRIGGER. When a doc records the behaviour you are changing as deliberate, say so and ask.
• Check git diff --cached --stat in a SEPARATE tool call, then commit by explicit paths. The tree is shared with other sessions.
• The commit message states what was wrong, what the evidence was, and what remains red. A fix whose measurement did not move says so.

Phase 6 — Amend this skill

This file is the only artefact that carries a lesson into the next session. Decide at the end of every run whether it should change and say which way — a silent skip is indistinguishable from a forgotten step.

Earns an edit: a cause Phase 0 would NOT have caught, that can recur (one bullet, dated); a rule you broke that cost real work; a signature here that proved wrong — correct it in place, keeping the retracted claim visible; a command, path or flag that changed under you.

Does not earn an edit: a one-off in one environment; a restatement of a rule already here; a finding about the PRODUCT rather than about diagnosing it; a run that went well.

Budget: stay under ~255 lines (240 → 250 → 255 on 2026-08-24, as dated signatures had to land in one run). When an addition would pass it, compress an existing item instead of appending — two bullets on one failure shape are one bullet. Commit the amendment with the work that produced it and say what was learned, not that the skill was updated.

Verification