Kurtosis Test

Run a local Kurtosis Ethereum testnet against erigon

This skill mirrors the CI workflow at .github/workflows/test-kurtosis-assertoor.yml but runs locally via the raw kurtosis CLI. The CI uses the ethpandaops/kurtosis-assertoor-github-action@v1 wrapper, which is not portable outside GitHub Actions; this skill drives kurtosis run, kurtosis enclave inspect, kurtosis service logs, and kurtosis enclave dump directly.

The skill takes an ethereum-package YAML config, builds the local test/erigon:current Docker image, starts a Kurtosis enclave, monitors EL/CL/assertoor/spamoor health, triages failures (challenging peer clients against erigon to identify the offender), and iterates a fix → rebuild → rerun loop until the testnet is stable or max-attempts is reached.

Inputs

The model parses these arguments and binds them to the shell variables used in the bash blocks below: $1/$2 are positional; duration=Nm → duration_secs, auto=true|false → auto, max-attempts=N → max_attempts.

Prerequisites

1. Docker running: docker info >/dev/null should succeed.
2. Kurtosis CLI installed: kurtosis version. Install from https://docs.kurtosis.com/install if missing. CLI ≥ 1.18.1 is only needed when the --package@branch you run includes the GpuConfig Starlark built-in (i.e. ethereum-package main post commit 835dd9b). The pinned branches in the mapping table below — including glamsterdam's 6.1.0 — predate that change, so they work on older CLIs. See Troubleshooting if you hit a GpuConfig Starlark error.
3. Erigon source tree at the cwd: Makefile exists and go.mod contains module github.com/erigontech/erigon.
4. curl and jq on $PATH — used by the monitor / triage snippets below to poll the EL JSON-RPC endpoint and the assertoor API.
5. Fork detection — skim the YAML for _fork_epoch keys. The repo's configs use the CL-side fork names: deneb_fork_epoch (Cancun on EL), electra_fork_epoch (Prague), fulu_fork_epoch (Osaka), gloas_fork_epoch (Amsterdam). Future fork keys will follow the same CL-naming convention. Feeds the spec-lookup section below.

Spec lookup (when debugging unfamiliar forks/EIPs)

If the YAML enables a fork under development, invoke /erigon-implement-eip Steps 2–4 to fetch:

• Step 2 — referenced/dependent EIPs.
• Step 3 — the meta EIP enumerating which EIPs the fork includes (CFI/SFI/PFI/DFI lists).
• Step 4 — the latest devnet specification at https://notes.ethereum.org/@ethpandaops/<devnet>.

Use these as ground truth when triaging. For specific opcodes / state transitions, also pull the EIP body via Step 1. If anything in the spec is contradictory or ambiguous, stop and ask the user rather than guessing — the same rule the EIP skill enforces.

Build the erigon docker image

The image tag must be exactly test/erigon:current because every .github/workflows/kurtosis/*.io config references that tag.

docker build -t test/erigon:current --build-arg BINARIES="erigon caplin" .

caplin is required in BINARIES because some configs (e.g. caplin-minimal-assertoor.io) use erigon as the CL via the same image.

Always rebuild before each run — the same approach the CI uses. BuildKit's layer cache makes the no-op rebuild fast, and the fix → rebuild → rerun loop necessarily picks up uncommitted source edits this way (a freshness check against git log would miss them and silently run a stale image).

If the user asks for a from-scratch binary build instead of docker, point them at /erigon-build; this skill itself uses docker because the kurtosis configs reference a docker image tag.

Suite → ethereum-package branch mapping (from CI)

The CI matrix pins different package branches per suite. Use the same pinning when the config matches a known CI file; for unknown configs, default to 5.0.1 and ask the user to confirm.

Note: glamsterdam is pinned to 6.1.0 rather than main because main introduced the GpuConfig Starlark built-in which requires kurtosis CLI ≥ 1.18.1. Caplin suites (caplin-assertoor.io, caplin-minimal-assertoor.io) require the erigontech fork — do not let them fall back to the default 5.0.1.

Start the testnet

ENCLAVE="${2:-kurtosis-test-$(date +%s)}"
CONFIG="$1"

# Map config basename → ethereum-package branch (mirrors the table above).
case "$(basename "$CONFIG")" in
  glamsterdam.io)
    PACKAGE_REF="github.com/ethpandaops/ethereum-package@6.1.0" ;;
  caplin-assertoor.io|caplin-minimal-assertoor.io)
    PACKAGE_REF="github.com/erigontech/ethereum-package@erigontech/fix-caplin-launcher" ;;
  regular-assertoor.io|pectra.io)
    PACKAGE_REF="github.com/ethpandaops/ethereum-package@5.0.1" ;;
  *)
    PACKAGE_REF="github.com/ethpandaops/ethereum-package@5.0.1" ;;
esac

kurtosis run \
  "$PACKAGE_REF" \
  --enclave "$ENCLAVE" \
  --args-file "$CONFIG" \
  --verbosity detailed --cli-log-level trace

Once kurtosis run returns, capture service names and host-mapped ports:

kurtosis enclave inspect "$ENCLAVE" --full-uuids

# Pick the first erigon EL service. `kurtosis enclave inspect` prints columnar
# rows (UUID first, then service name), so we match against field 2.
EL_SERVICE=$(kurtosis enclave inspect "$ENCLAVE" --full-uuids 2>/dev/null \
  | awk '$2 ~ /^el-[0-9]+-erigon-[a-z]+$/ {print $2; exit}')
EL_RPC_PORT=$(kurtosis port print "$ENCLAVE" "$EL_SERVICE" rpc 2>/dev/null \
  | sed -E 's|.*:([0-9]+).*|\1|')

# CL endpoint (whichever client paired with that EL)
CL_SERVICE=$(kurtosis enclave inspect "$ENCLAVE" --full-uuids 2>/dev/null \
  | awk '$2 ~ /^cl-[0-9]+-[a-z]+-erigon$/ {print $2; exit}')
CL_HTTP_PORT=$(kurtosis port print "$ENCLAVE" "$CL_SERVICE" http 2>/dev/null \
  | sed -E 's|.*:([0-9]+).*|\1|')

# Optional services
ASSERTOOR_PORT=$(kurtosis port print "$ENCLAVE" assertoor http 2>/dev/null | sed -E 's|.*:([0-9]+).*|\1|')
DORA_PORT=$(kurtosis port print "$ENCLAVE" dora http 2>/dev/null | sed -E 's|.*:([0-9]+).*|\1|')
SPAMOOR_PORT=$(kurtosis port print "$ENCLAVE" spamoor http 2>/dev/null | sed -E 's|.*:([0-9]+).*|\1|')

Print the assertoor / dora URLs so the user can open the dashboards in a browser.

Monitor

Three checks run on a polling loop until either (a) duration elapses, or (b) any failure trips. All three are captured in the run history.

Check A — Block height progress

duration_secs is the parsed duration=Nm input (default 1200). Three terminal outcomes:

• STABLE — chain produced blocks and the duration elapsed without a stall.
• STALL — chain produced at least one block, then stopped advancing for >3 × seconds_per_slot.
• NO_PROGRESS — chain never produced a block within duration_secs (e.g. validators didn't start).

duration_secs=${duration_secs:-1200}
prev=0
slot_secs=$(grep -E '^\s*seconds_per_slot:' "$CONFIG" | awk '{print $2}'); slot_secs=${slot_secs:-12}
poll_interval=$(( slot_secs * 2 ))
stall_window=$(( slot_secs * 3 ))
start=$(date +%s)
end=$(( start + duration_secs ))
stall_deadline=$(( start + stall_window ))
outcome=""

while [ "$(date +%s)" -lt "$end" ]; do
  height_hex=$(curl -s --max-time 5 "http://127.0.0.1:${EL_RPC_PORT}" \
    -H 'Content-Type: application/json' \
    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    | jq -r '.result // empty')
  if [[ "$height_hex" =~ ^0x[0-9a-fA-F]+$ ]]; then
    height=$(printf '%d\n' "$height_hex")
    echo "[$(date -u +%H:%M:%S)] height=$height"
    if [ "$height" -gt "$prev" ]; then
      prev=$height
      stall_deadline=$(( $(date +%s) + stall_window ))
    fi
  else
    echo "[$(date -u +%H:%M:%S)] RPC unreachable or invalid response — retrying"
  fi
  # Only declare a stall after seeing at least one block. Many configs set
  # genesis_delay > stall_window (e.g. glamsterdam.io: 20s delay vs 18s window
  # at 6s slots), so the pre-genesis gap would otherwise trip a false stall.
  if [ "$prev" -gt 0 ] && [ "$(date +%s)" -gt "$stall_deadline" ]; then
    outcome="STALL: chain not progressing for >${stall_window}s (last height=$prev)"
    break
  fi
  sleep "$poll_interval"
done

if [ -z "$outcome" ]; then
  if [ "$prev" -eq 0 ]; then
    outcome="NO_PROGRESS: chain never produced a block within ${duration_secs}s"
  else
    outcome="STABLE: chain progressed for full ${duration_secs}s window (final height=$prev)"
  fi
fi
echo "$outcome"

Pass: height advances ≥1 within every 3 × seconds_per_slot (the stall window), sustained for the full duration_secs (poll cadence is 2 × seconds_per_slot). Fail: STALL or NO_PROGRESS.

Check B — Assertoor results

curl -s "http://127.0.0.1:${ASSERTOOR_PORT}/api/v1/test_runs" \
  | jq '.data[] | {name, status, result}'

Pass: every test_run has result=success. Fail: any result=failure, or any test stuck pending / running past 3× its expected duration. The assertoor web UI at http://127.0.0.1:${ASSERTOOR_PORT}/ shows per-step trees; use it for deep dives.

Check C — Erigon-focused log scan

kurtosis service logs "$ENCLAVE" "$EL_SERVICE" 2>&1 \
  | grep -iE 'panic|fatal|^ERROR|"lvl"="error"|consensus failure|invalid block' \
  | tail -200

For cross-client comparison (used by the triage section), run the same scan across every EL/CL service:

for svc in $(kurtosis enclave inspect "$ENCLAVE" --full-uuids \
             | awk '$2 ~ /^(el|cl|vc)-[0-9]+-[a-z]+-[a-z]+$/ {print $2}'); do
  echo "=== $svc ==="
  kurtosis service logs "$ENCLAVE" "$svc" 2>&1 \
    | grep -iE 'error|panic|fatal' | tail -30
done

If snooper-engine-* services exist (when snooper_enabled: true in the YAML), pull their logs too — they capture the full Engine API request/response trace, invaluable when an EL bug is suspected.

Issue detection criteria

A single failed check trips the triage section. Block-stall + erigon panic + assertoor fail are independent signals — record all three in the run history; do not stop at the first.

Debugging methodology — triage erigon vs peer-client vs network/config

Decision tree:

1. Reproduce. A single one-shot failure gets one re-run before triaging. Truly intermittent failures still get triaged, but flag them as flaky.
2. Classify the symptom. One of: block-stall, EL panic, EL invalid-payload, CL fork-choice mismatch, assertoor opcode/EIP test failure, spamoor tx-submission failure.
3. Cross-client comparison. For each erigon-side error, find the equivalent moment in the peer-client log at the same slot/block. Three outcomes:
   • Erigon wrong: erigon rejects/panics; peer client + assertoor accept the block → erigon bug, fix locally.
   • Peer wrong: erigon accepts; peer rejects → check peer-client image tag against the fork's expected tag (often a stale image). Surface to user; do not fix erigon.
   • Both disagree with spec: clients produce different "valid" answers from what the EIP spec says → escalate to the user. Likely spec ambiguity or a misread.
4. Cross-reference the spec. Pull the relevant EIP (/erigon-implement-eip Step 1) and the devnet spec (/erigon-implement-eip Step 4) for the failing block, opcode, or state transition.
5. Rule out config drift. Diff the YAML's el_extra_params, network_params, and fork epochs against the equivalent CI suite under .github/workflows/kurtosis/. Mismatches there are config bugs, not erigon bugs.
6. Rule out enclave plumbing. kurtosis service exec <enclave> <svc> "ping <other_svc>" to verify network reachability; check JWT mounting via kurtosis service exec <enclave> <el-svc> "ls -la /jwt/". The CLI takes the command as a single positional arg (multi-word commands must be quoted) — there is no -- separator.

Triage table

Fix-rebuild-rerun loop

Auto-iterates by default (auto=true), capped at max-attempts=5. Per attempt:

1. Tear down: kurtosis enclave rm -f "$ENCLAVE".
2. Apply the fix to erigon source via Edit. Only auto-apply when the triage classified the issue as "Erigon wrong" with high confidence. If ambiguous (peer could be wrong, or spec interpretation unclear), pause and surface to the user even before the cap — that overrides auto=true.
3. Rebuild image: docker build -t test/erigon:current --build-arg BINARIES="erigon caplin" ..
4. Re-launch: same config, fresh timestamped enclave name (so each attempt's dump stays separate).
5. Re-run monitor: same three checks.
6. Record: per attempt — symptom, hypothesis, fix applied, outcome.

After max-attempts consecutive failures, halt and print the per-attempt history. Do not auto-apply more fixes once the cap is hit. Reference: /autoresearch follows the same iterate-and-record pattern.

If auto=false, pause for user approval before steps 2–4 of every attempt.

Cleanup (always run)

Run as the final step of every iteration regardless of outcome (success, failure, or user interrupt):

DUMP_DIR="/tmp/kurtosis-dump-${ENCLAVE}"
# `kurtosis enclave dump` refuses if the destination already exists — never pre-create it.
kurtosis enclave dump "$ENCLAVE" "$DUMP_DIR" || true
kurtosis enclave rm -f "$ENCLAVE" || true
echo "Logs dumped to: $DUMP_DIR"

The dump contains per-service logs (el-*, cl-*, vc-*, assertoor, spamoor, dora, snooper-*) — keep this directory until triage is complete; GitHub blob storage is not in play here so the logs are only on disk locally.

After multiple iterations, also prune dangling docker images:

docker image prune -f

Troubleshooting