# Test Sh Monitor

> Guide for running and monitoring `dev-support/test.sh` in conflux-rust. Use this skill whenever the user wants to run tests, launch test.sh, monitor test progress, check test results, set up a new worktree for testing, or diagnose test failures in the conflux-rust project. Also trigger when the user asks about test phases, log keywords, build failures, or integration test failures in this repo.

- Skill: `conflux-chain/test-sh-monitor` (Agent Skill)
- Install (CLI): `npx skillmds@latest add conflux-chain/test-sh-monitor`
- Raw SKILL.md: https://api.skillmd.com/api/skills/conflux-chain/test-sh-monitor/raw
- Safety review: WARNING
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: conflux-chain (https://skillmd.com/u/conflux-chain)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/conflux-chain/test-sh-monitor

---


# test.sh Running and Monitoring Guide

## Core Principle

**Always run the complete `test.sh` — never execute individual phases in isolation.**

`test.sh` sets critical environment variables (`CARGO_TARGET_DIR`, `RUSTFLAGS`, `CONFLUX_BENCH`), activates the venv, and manages symlinks. These side effects only take effect during a full run. In particular, `CARGO_TARGET_DIR=$ROOT_DIR/build` exists only within the `test.sh` process environment — manually running `cargo build` inside `tools/consensus_bench/` will drop artifacts in the wrong location, causing integration tests to pick up a stale binary.

---

## Step 0: Ask for Polling Interval

Before doing anything else, ask the user:

> "How often should I check the log? Options: **60s**, **120s**, or a custom value."

Remember the chosen interval — it will be used in the monitoring loop in Step 3.

---

## Step 1: Pre-flight Checks

Confirm these five items before launching to avoid discovering environment problems after a long wait:

```bash
# 1. uv is present
command -v uv || echo "MISSING: run: curl -LsSf https://astral.sh/uv/install.sh | sh"

# 2. submodules are initialized (required separately for each new worktree)
git submodule status | grep "^-" && echo "MISSING submodules: run git submodule update --init --recursive" || echo "submodules OK"

# 3. target directory state (must be a symlink, not a real directory)
ls -la target 2>/dev/null || echo "target not present (OK, test.sh will handle)"

# 4. venv (test.sh creates it automatically; this is just a sanity check)
ls .venv 2>/dev/null && echo "venv exists" || echo "venv absent (test.sh will create it)"

# 5. fmt check (fast — catches formatting issues before starting the long build)
bash cargo_fmt.sh -- --check
```

If fmt check fails, fix formatting before launching test.sh — there is no point waiting for a build that will eventually fail the same check.

If submodules are missing:
```bash
git submodule update --init --recursive
```

---

## Step 2: Launch test.sh

**Prefer `run_in_background: true`** so the framework owns the process and notifies you on exit, structurally eliminating zombie process issues:

```bash
bash -c 'set -o pipefail; bash dev-support/test.sh 2>&1 | tee /tmp/test_run.log'
# (with run_in_background: true)
```

> **Why pipe instead of `> file 2>&1`?** test.sh's check functions use `tee /dev/stderr` internally. When stderr is redirected to a regular file, `tee` opens `/dev/stderr` via `/proc/self/fd/2`, which creates a new file description starting at position 0 — overwriting earlier content (including phase anchors). With a pipe, there is no file position, so `tee /dev/stderr` appends correctly. `pipefail` ensures the exit code reflects test.sh, not tee.

> **Why prefer run_in_background?** The framework-owned process is not a child of the current shell, so the framework reaps it — no need to handle zombies in the monitoring loop.

If you must background it manually, **do not use `kill -0` as the loop condition** — it cannot distinguish a running process from a zombie (`kill -0` returns 0 for both). Use `ps -o stat=` instead:

```bash
bash -c 'set -o pipefail; bash dev-support/test.sh 2>&1 | tee /tmp/test_run.log' &
PID=$!
while [[ "$(ps -p $PID -o stat= 2>/dev/null)" =~ ^[^Z] ]]; do
    sleep 10
    tail -5 /tmp/test_run.log
done
wait $PID
echo "exit: $?"
tail -50 /tmp/test_run.log
```

`ps -o stat=` returns a status character: `S`/`R`/`D` = running normally, `Z` = zombie, empty = process gone. The regex `^[^Z]` exits the loop on either zombie or gone; `wait $PID` then reaps and returns the true exit code.

---

## Step 3: Periodic Monitoring

The core check logic is the same regardless of tool availability:

```bash
PASSED=$(grep -c "✓" /tmp/test_run.log 2>/dev/null || echo 0)
FAILED=$(grep -cE "[✖✗]" /tmp/test_run.log 2>/dev/null || echo 0)
RUNNING=$(pgrep -f "bash dev-support/test.sh" > /dev/null 2>&1 && echo "RUNNING" || echo "STOPPED")
PHASE=$(grep -oE "(Phase [0-9]/[0-9]: [^=]+|Fmt check( passed)?)" /tmp/test_run.log 2>/dev/null | tail -1 || echo "unknown")
echo "[$(date '+%H:%M:%S')] $RUNNING phase=[$PHASE] passed=$PASSED failed=$FAILED"
tail -3 /tmp/test_run.log
```

Keep the check logic simple — count passes/fails, detect phase from anchors, tail the log. Do NOT attempt smart phase inference in the script; leave interpretation to the AI.

Below are two options depending on tool availability. **Prefer Option A** if the Monitor tool is available — it runs the polling loop as a background task whose stdout lines become chat notifications, eliminating the manual alternating pattern. Option B is the fallback for Claude Code versions that do not have the Monitor tool.

### Option A: With Monitor tool

Use Monitor as a log filter. The script runs a polling loop; each `echo` becomes a notification. Only emit output on noteworthy events — phase changes, new failures, or process exit — to avoid flooding the conversation:

```bash
Monitor({
  description: "test.sh watchdog",
  timeout_ms: 3600000,
  persistent: false,
  command: """
PREV_PHASE=""
PREV_FAILED=0
while true; do
  if [ ! -f /tmp/test_run.log ]; then sleep 10; continue; fi
  RUNNING=$(pgrep -f "bash dev-support/test.sh" > /dev/null 2>&1 && echo "yes" || echo "no")
  PASSED=$(grep -c "✓" /tmp/test_run.log 2>/dev/null || echo 0)
  FAILED=$(grep -cE "[✖✗]" /tmp/test_run.log 2>/dev/null || echo 0)
  PHASE=$(grep -oE "(Phase [0-9]/[0-9]: [^=]+|Fmt check( passed)?)" /tmp/test_run.log 2>/dev/null | tail -1)

  if [ "$PHASE" != "$PREV_PHASE" ]; then
    echo "[$(date '+%H:%M:%S')] $PHASE | passed=$PASSED failed=$FAILED"
    PREV_PHASE="$PHASE"
  fi
  if [ "$FAILED" -gt "$PREV_FAILED" ]; then
    echo "[$(date '+%H:%M:%S')] NEW FAILURE: failed=$FAILED (was $PREV_FAILED)"
    grep -E "[✖✗]" /tmp/test_run.log 2>/dev/null | tail -5
    PREV_FAILED=$FAILED
  fi
  if [ "$RUNNING" = "no" ]; then
    echo "[$(date '+%H:%M:%S')] EXITED | passed=$PASSED failed=$FAILED"
    tail -10 /tmp/test_run.log
    exit 0
  fi
  sleep <interval>
done
"""
})
```

### Option B: Without Monitor tool

Use the foreground Bash + background sleep alternating pattern:

**Step A — foreground log check (no sleep):**

```bash
PASSED=$(grep -c "✓" /tmp/test_run.log 2>/dev/null || echo 0)
FAILED=$(grep -cE "[✖✗]" /tmp/test_run.log 2>/dev/null || echo 0)
RUNNING=$(pgrep -f "bash dev-support/test.sh" > /dev/null 2>&1 && echo "RUNNING" || echo "STOPPED")
PHASE=$(grep -oE "(Phase [0-9]/[0-9]: [^=]+|Fmt check( passed)?)" /tmp/test_run.log 2>/dev/null | tail -1 || echo "unknown")
echo "[$(date '+%H:%M:%S')] $RUNNING phase=[$PHASE] passed=$PASSED failed=$FAILED"
tail -3 /tmp/test_run.log
```

**Step B — background timer (run_in_background: true):**
```bash
sleep <interval chosen in Step 0>
```

On timer notification → run Step A → run Step B → repeat.

> **Why this pattern?** The foreground Bash call returns immediately without blocking the conversation; every report is visible in real time. The background sleep is purely a timer. Common mistakes: background bash loop (intermediate output invisible), foreground sleep (blocks the conversation), background Agent for polling (same problem as background bash loop).

---

## Phases

### Pre-check: Fmt
- **Anchor:** `=== Fmt check ===`
- **Success anchor:** `=== Fmt check passed ===`
- **Behavior:** runs `cargo_fmt.sh -- --check` (nightly rustfmt across all workspaces). Fast (seconds). On failure, process exits immediately — no success anchor, no build phases.
- **Note:** Step 1 pre-flight already runs this check independently. If you see fmt failure here, it means the pre-flight was skipped or code changed between pre-flight and launch.

### Phase 1: cargo build (main project)
- **Anchor:** `=== Phase 1/4: Building main project ===`
- **Log signal:** continuous `Compiling xxx` (from cargo stderr; absent when build is cached)
- **Success anchor:** `=== Phase 1/4: Build succeeded ===`
- **Failure:** `error[E...]` from cargo stderr — process exits immediately (no success anchor)

### Phase 2: cargo build consensus_bench
- **Anchor:** `=== Phase 2/4: Building consensus_bench ===`
- **Success anchor:** `=== Phase 2/4: Build succeeded ===`

### Phase 3: test_all.py (integration tests) — requires active monitoring
- **Anchor:** `=== Phase 3/4: Integration tests ===`
- **Key behavior:** parallel scheduling; a single test failure **does not** exit the process. Early break triggers: (1) >5 failures in a single round, or (2) any test that fails twice across retry rounds
- **Process alive ≠ tests passing** — tail the log regularly and count `✖`
- **Success anchor:** `=== Phase 3/4: Integration tests passed ===`
- **Failure:** any `✖`; exit code 1 or 80

### Phase 4: pytest
- **Anchor:** `=== Phase 4/4: Pytest ===`
- **Watch for:** `FAILED` and `ERROR` (pytest uses these, not `✖`)
- **Success anchor:** `=== Phase 4/4: Pytest passed ===`

---

## Log Keyword Reference

| Keyword | Phase | Meaning | Action |
|---------|-------|---------|--------|
| `=== Phase N/4:` | any | Phase anchor from test.sh | Reliable phase indicator |
| `Compiling` | 1/2 | Cargo compiling (stderr) | Normal; absent when cached |
| `error[E` | 1/2 | Rust compile error (stderr) | Read full log immediately |
| `Scanning num_nodes` | 3 | test_all.py starting | Build phases completed |
| `✓` | 3 | Single integration test passed | Normal |
| `✖` | 3 | Single integration test failed | Count; >5 means likely failure |
| `PASSED` | 4 | Single pytest passed | Normal |
| `FAILED` | 4 | Single pytest failed | Investigate |
| `The following test fails:` | 3 | Integration test summary | List of all failed tests |
| `ModuleNotFoundError` | any | Python env not ready | Stop, diagnose venv |
| `Cannot found contract` | 3 | Submodule not initialized | `git submodule update --init --recursive` |
| `externally-managed-environment` | startup | PEP 668, pip blocked | Check uv install; see activate_new_venv.sh |

**Not in the log:** `Build succeeded.` / `Build failed.` / `Integration test failed.` / `Pytest failed.` — these strings are captured into shell variables by `$()` and never printed to stdout/stderr. Use the `=== Phase ===` anchors instead.

---

## New Worktree Quick Start

When the current branch is already checked out by the main worktree, use `--detach` to avoid the branch conflict:

```bash
# 1. Create worktree
git worktree add --detach /home/ubuntu/worktrees/my-test HEAD

# 2. Initialize submodules (required for every new worktree)
cd /home/ubuntu/worktrees/my-test
git submodule update --init --recursive

# 3. (Optional) Reuse build artifacts to skip 30–50 min of compilation
#    Requires both worktrees to be on the same commit
ln -s /conflux-rust/build /home/ubuntu/worktrees/my-test/build

# 4. Launch (run_in_background: true)
bash -c 'set -o pipefail; bash dev-support/test.sh 2>&1 | tee /tmp/test_run.log'
```

---

## Three Monitoring Layers — All Required

1. **Process layer:** is the process still alive? (compile failures exit quickly)
2. **Log layer:** tail regularly and interpret keywords by phase
3. **Semantic layer:** in Phase 3, "process alive" does not mean "tests passing" — check the `✖` count