# Dual Instance Verification

> Dual-Instance Verification (双机实测)

- Skill: `org2ai/dual-instance-verification` (Agent Skill)
- Install (CLI): `npx skillmds@latest add org2ai/dual-instance-verification`
- Raw SKILL.md: https://api.skillmd.com/api/skills/org2ai/dual-instance-verification/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: org2ai (https://skillmd.com/u/org2ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/org2ai/dual-instance-verification

---


# Dual-Instance Verification (双机实测)

Real-machine verification of session sharing across ORG2 (primary, Neonforge) and
ORG2 Instance 2 (VantaNode). Born from a four-bug escape on 2026-07-24 where every
bug passed the old three-piece check (resource curves + feature signals +
WARN/ERROR delta). The disciplines below exist because each one, applied that day,
would have caught at least one escaped bug.

## Core principle

**Assert invariants, not absence of errors.** A scenario passes only when the
positive end-state is proven on THREE surfaces — sender instance, receiver
instance, and the cloud rows — and every state mutation in between is explainable.

## Non-negotiables

1. **Cloud ground-truth ledger — fleet-wide, invariant-based.** Snapshot
   `cloud_sessions` (session_id, deleted_at, access_mode, events_count,
   events_frozen_seq, events_epoch, stored_bytes) BEFORE and AFTER every
   scenario, via service key — for EVERY org the instances can see, not just the
   org under test. Diff must be explainable line-by-line, and "explainable"
   means a verified mechanism, not a plausible story ("that session is active"
   is a story; "its rollout grew by N lines, here they are" is a mechanism).
   On top of the diff, assert invariants: for every session the scenario did
   NOT deliberately touch, `events_epoch` is CONSTANT and `events_count` is
   monotone; flag any row with `events_epoch` above a small threshold (>3)
   anywhere in the fleet. Any unexplained `deleted_at`, access_mode downgrade,
   events_count drop, or epoch bump is a FAILURE even if the UI looks fine.
   (Would have caught: vanished-sweep mass retract, boot out-of-scope retract,
   and the #608 rewrite storm — a 28-epoch counter sat in this column for weeks
   while diffs on the test org alone stayed clean.)

2. **Destructive-effect audit at INFO level — classify by effect, not verb.**
   After each scenario AND after each app boot, grep both instances'
   frontend+backend logs for
   `retract|untag|drop|delete|demote|evict|vanish|superseded|epoch rewrite|rewrite`
   at ALL levels, not just WARN/ERROR. Every hit needs a justification. The
   audit's unit is "anything that replaces or deletes cloud bytes" — a full
   epoch rewrite re-uploads and REPLACES the entire stored copy and is more
   destructive than a retract, yet it wears routine INFO wording and matches no
   scary verb. When new log lines gain the power to mutate cloud rows, add them
   to this list in the same PR.

3. **Lifecycle-boundary cells are mandatory.** The matrix is feature ×
   lifecycle-event, not feature × instance. For every sharing feature, run at
   minimum these transition cells:
   - **Cold boot**: relaunch each instance, then audit the FIRST 3 sync passes'
     cloud mutations (ledger diff + verb audit). Boot-window races (empty scope
     mirror, unrefreshed token, rebuilding cache) must never be treated as
     authoritative absence.
   - **/compact mid-share**: continue a shared session into a continuation
     sibling; the cloud row must survive (not retract) and sharing must carry on.
   - **Cache wipe/rebuild**: wipe imported-history cache, boot, confirm zero
     retracts during rebuild (two-strike must defer).
   - **Fork-on-write**: owner-side AND guest-side, with a real (small) agent run.
     Assert the fork's cloud row access_mode equals the inherited level, events
     and frozen segments actually land, and the receiver can OPEN the replay
     (bright row + content renders). "Row appears in the list" is NOT success —
     a metadata-only ghost also appears.

4. **Watchdog/fallback fires are test failures.** Any
   `usePlanningIndicator watchdog`, dead-man, forced-idle, or retry-exhausted
   line during a scenario fails that scenario, even though the UI self-heals.
   Self-healing masks the bug it recovers from. Turn-completion latency must be
   asserted: terminal reaches the UI within 5s of Rust `state=Completed`.

5. **Receiver-depth assertions.** On the receiving instance: open the shared/fork
   row, confirm content renders, and for replay confirm the latest round matches
   the sender's last round verbatim. Grayed-out rows must be explained by an
   asserted access mode, never shrugged off.

6. **Both directions.** Each cell runs A→B and B→A where roles permit. Guest-side
   limitations (e.g. no API key on inst2) mean the cell moves to the other
   instance, not that it gets skipped. If a cell is skipped for cost, record it as
   UNCOVERED in the delivery message — the 2026-07-24 fork bugs lived in exactly
   such a silently skipped cell.

7. **Silent early-returns need a diagnostic.** When touching sync/channel/handler
   code: any `return` that swallows a lifecycle-relevant frame or skips a push
   must have a rate-limited log. The four-bug escape survived five instrumentation
   builds only because `bus dispatch`, `waitForSessionChannelReady`,
   `routeSessionChannelEvent`, `handleEvent(_disposed)`, and the runtime-status
   gate all dropped silently. Those five now log; keep that bar for new code.

8. **Resource three-piece stays — and anomalies are defects until mechanized.**
   cmd+5/Activity Monitor curves (idle ≈0%, RSS returns to baseline), feature
   signals, and WARN/ERROR delta with each new line triaged. This skill ADDS to
   it; it does not replace it. A recurring resource anomaly (a CPU wave on
   every boot, RSS that climbs per pass) must open an investigation cell — it
   may NOT be closed with a narrative. The #608 storm's boot-time CPU wave was
   observed, named "ingest re-hash convergence", and normalized; the re-hash
   WAS the bug. Naming an anomaly is not explaining it.

## Invariant & determinism cells (mandatory additions per run)

Born from the 2026-07-30 reflection on why #608 (rewrite storm), the hollow
wipe, and the scope flap all survived multiple live rounds: the protocol
asserted presence (the tested flow works) while these bugs were silent surplus
actions in the background, invisible to every existing cell.

- **Two-boot determinism cell.** With local state unchanged, cold-boot the
  instance twice and let sync passes run. Boot 2 must produce ZERO epoch
  rewrites and zero destructive-effect hits (boot 1 may re-anchor once after a
  legitimate format/order change — each such rewrite must be explained as
  exactly-once). Nondeterminism bugs are per-process (HashMap iteration order,
  random seeds); a single boot cannot sample them by construction. Measured
  cost: ~18 minutes wall clock, mostly unattended.
- **Absence needs liveness beside it.** Any "zero X since the fix" claim must
  pair the constant (epoch, deleted_at) with a mover (events_count,
  updated_at, pass counters in the log) proving the engine actually ran over
  the rows in question. Deferred/skipped sessions (e.g. scope-guard deferrals)
  are UNCOVERED, not passing.
- **At least one fault-injection cell per run.** Healthy instances sample only
  the happy path; the hollow wipe (empty local read while cursor covers >0)
  and the scope flap (transient GitHub identity failure) lived exclusively in
  degraded states no healthy-path cell can reach. Rotate through: rename/move
  a local source DB mid-run (hollow read), block the identity endpoint
  (lookup failure), kill the app mid-transfer (partial persist). The guard
  under test must defer/refuse — any destructive act under injected fault is
  a failure. When the change ADDS a fault point (a new read, probe, or IPC
  call), inject THAT fault — the rotation list only covers yesterday's
  failure modes.
- **Upgrade cell: persisted state must cross the version boundary.** Any
  change that reads durable state written by earlier builds (push cursors,
  cache metadata, parser output, settings) gets one cell where the OLD build
  writes the state and the NEW build operates on it: run a pre-change binary
  (a dated `org2-main.exe` or a develop build) through the flow first, then
  swap binaries over the SAME homes and continue. Assert the new build rides
  the ordinary incremental path — no epoch rewrite, no refuse, no silent
  re-derive — and that a second cycle (new build writes, new build reads)
  is idempotent. A fresh-anchor test with only the new binary proves nothing
  about migration: PR #692's costliest bug (every legacy flat cursor forced
  an O(total) epoch rewrite) was invisible to every run that built its state
  with the new code. Second-order cycles count too: state the new build
  STAMPS must survive the new build's own next scan/rescan before the
  invariant is real (PR #693's lineage stamp was erased by the very next
  rescan's metadata rewrite).
- **Unexplained delta becomes a cell.** The first ledger delta, log line,
  resource pattern, or store-vs-UI discrepancy without a mechanism-level
  explanation is promoted to a scenario in the CURRENT run — not noted for
  later, and not handed to a background task. "Background sync noise" is the
  phrase that hid a data-destroying storm; "pre-existing behavior" is the
  phrase that hides everything the current PR did not happen to cause.
- **Baseline A/B answers attribution, not existence.** Reproducing a symptom on
  the develop baseline proves the PR under test did not CAUSE it. It proves
  nothing about whether it is a bug, and it is not a disposition. Write the two
  conclusions on separate lines — attribution (this PR / not this PR) and
  verdict (defect / expected, with the mechanism) — and never let the first
  supply the second. A symptom that survives A/B is either explained
  mechanically in this run or recorded as an OPEN DEFECT with its evidence in
  the delivery message; deferring it to a chip is the same escape as "noted for
  later" above. Corollary signature: **the local store holds N rows and the UI
  renders 0** is always a defect — chase it to the command and the filter that
  ate the rows before moving on, because the two ends disagreeing is itself the
  mechanism-level question. (2026-08-01: instance-2's sidebar OLDER section
  rendered zero imported rows while `imported_history_session_cache` held 257
  cursor_ide rows. A/B correctly cleared PRs #628/#576 of causing it; the
  symptom was then downgraded to a background chip and is STILL unexplained as
  of 2026-08-03, including after the unrelated #654 import-parse regression was
  ruled out as its cause.)

## Ledger commands

Service key lives in `tests/e2e/.env` (machine-local). Snapshot:

```bash
set -a; source tests/e2e/.env; set +a
curl -s "$E2E_CLOUD_SUPABASE_URL/rest/v1/cloud_sessions?org_id=eq.<ORG>&select=session_id,deleted_at,access_mode,events_count,events_frozen_seq,stored_bytes,updated_at&order=session_id" \
  -H "apikey: $E2E_CLOUD_SERVICE_KEY" -H "Authorization: Bearer $E2E_CLOUD_SERVICE_KEY" \
  -H "Accept-Profile: org2_cloud"
```

Diff the before/after JSON; explain every changed row. Logs live at
`~/.orgii/logs/` and `~/.orgii-instance2/logs/` — backend files are UTC-dated and
UTC-stamped, frontend files local-stamped; sweep BOTH around the UTC midnight
rollover or the window silently truncates.

## Failure taxonomy (what escaped and why — keep this list growing)

- **Boot-window absence treated as authority**: empty scope mirror / rebuilding
  cache / unrefreshed token read as "gone" → retract. Guard: grace period or
  two-strike before any destructive act on boot-adjacent passes.
- **Continuation demotion read as deletion**: /compact demotes the old sibling;
  exact-id lookups report it absent by design; sweeps must use the
  superseded-inclusive lookup.
- **Defaults silently degrading shares**: a fork with no sharing-ladder entry
  floors to metadata_only and nobody errors. Assert access_mode on the wire.
- **Self-healing hiding lost signals**: watchdog-forced completion masked every
  lost agent:complete. Assert latency, treat watchdog as failure.
- **A guard upstream of every probe**: the subagent bridge swallowed fork
  terminals before any instrumented drop point ran, so nine instrumented
  builds all stayed silent. When probes disagree with the symptom, suspect the
  model of WHERE the loss happens, not a missed branch — walk the call path
  from its first line, not from the suspected failure.
- **Format drift on a shared field**: `Session.orgId` is a scope selector
  (`cloud:<uuid>`); fork/import wrote a bare uuid, silently removing every
  ownership-derived affordance. When one writer of a shared field disagrees
  with the rest, diff the live values across rows — the odd one out is the
  bug.
- **Tests that encode the bug**: the two specs guarding the ownership stamp
  asserted the bare form, comment included. Green tests are not evidence the
  convention is right; check a spec's expectation against the consumers before
  trusting it.
- **Fixture writes that silently failed**: a direct PATCH on a
  write-hardened table (403 — governance requires the admin RPC) surfaced as
  an empty response body, was read as success, and a whole debugging night ran
  on the false premise that the scope existed server-side. Every mutation of
  test-environment cloud state MUST be followed by a read-back of the same
  row (compare `updated_at`, not just the field). A stale local mirror is not
  server truth either — when a UI decision depends on mirrored state, diff
  mirror vs server before blaming the consumer code.
- **The running binary lags the fix**: the "verified" build predated the
  final commit of the file under test; every on-device probe for 40 minutes
  exercised stale code. Before declaring an on-device verdict, compare the
  bundle mtime against the fix file's mtime — an edit made after the last
  build is not on the device, no matter how green the tests are.
- **A feature unreachable from the surface it was designed for**: Address
  Comments' run path is fork-first BY DESIGN for imported histories, but that
  composer mounts session-scope "none", and every consumer in the chain
  (slash registry, submit interceptor) re-resolved the blank id and silently
  no-opped. Reachability must be verified from the surface the design names.
  Same family: candidate ordering picked a scope-matching org with no server
  row (GitHub rename made two spellings one repo network), and the fork guard
  demanded snapshot == summary while a LIVE source kept growing — equality
  checks against a moving target are boot-window absence in another costume.
- **A silence that proves nothing**: "zero rewrites since the fix" was true
  while the session was not being pushed at all (machine slept, ingest
  follows the open view). An absence metric needs a liveness metric beside
  it: assert the thing you want CONSTANT (ledger epoch) against the thing
  that must still be MOVING (events_count / updated_at). Same shape as
  watchdog-masked completion — silence and health look identical until you
  measure both.
- **Presence oracles miss surplus actions**: every cell asserted "the thing I
  did worked"; the escaped bugs were things NOBODY did — silent rewrites,
  silent retracts, a silent wipe — that break no foreground flow. The fleet
  ledger's invariant columns (epoch constant, count monotone) are the only
  surface where surplus actions are visible at all.
- **Per-process nondeterminism is invisible to single-boot runs**: HashMap
  iteration order reshuffled an unchanged transcript's flushed tail on every
  boot, and positional chunk ids turned the shuffle into a fresh hash chain
  each time. Within one app lifetime everything looked stable; only
  boot-vs-boot comparison of push decisions could see it. (#608 root cause.)
- **Fresh-state runs cannot see migration bugs**: every cell that builds its
  own state with the binary under test samples only the post-change state
  space. Bugs that live in the TRANSITION — legacy cursor meets new hash
  mode, old parser rows meet new election, stamped metadata meets the next
  rescan — need the upgrade cell above. The tell is a verification report
  whose every artifact was created during the run itself.
- **"Pre-existing" used as a verdict**: a symptom reproduces on baseline, is
  correctly cleared of THIS PR's authorship, and is then silently cleared of
  being a bug at all — because the run's attention is scoped to the PR, and
  attribution is the only question the run was asking. Nothing in the protocol
  catches this: every other discipline here fires on something the run DID,
  while this one fires on a verdict the run declined to reach. The tell is a
  finding whose write-up names the PR it exonerates but never names a
  mechanism.
- **Waiting for a pass the engine will never run**: the session plane follows
  visible-org demand — an org's push/retract pass runs only while that org is
  the active workspace. A fix whose cleanup rides "the next pass" looks
  broken for any org you are not looking at. Per-org verification must OPEN
  the org (switch the workspace to it) as the trigger, and cleanup claims
  must name which orgs were actually visited. Corollary: rows pushed in an
  earlier install/test cycle may have no surviving local push-state, and the
  client rightly refuses to retract what it cannot prove it pushed — those
  need a server-side fixture sweep, not more waiting.

## When NOT to use

Pure UI styling/copy changes, single-file bug fixes with no sync surface, and
Rust-only refactors covered by unit tests that touch no cloud or channel path.