Run Network Benchmark

Run the apollo network benchmark

apollo_network_benchmark spins up N libp2p nodes that broadcast stress-test messages to each other and measures delivery latency, throughput, and message loss. Everything is driven by one orchestrator binary:

cargo run --release --bin apollo_network_benchmark_run -- <local|cluster> <start|stop|logs|port-forward> [flags]

• local: Docker Compose on this machine — one container per node plus Prometheus (:9090), Grafana (:3000), and cAdvisor (:8080). Best for quick runs.
• cluster: GKE on sequencer-dev (us-central1) using an Indexed Job on the apollo-network-benchmark node pool. Best for scale and isolated resources.

Crate: crates/apollo_network_benchmark. The orchestrator builds the Docker image, generates configs, and deploys — you never invoke the node binary directly.

Step 1: Gather parameters

Ask ONE AskUserQuestion round. Skip any question the user already answered in their prompt — map their words onto flags using the reference table below. Everything not asked uses defaults.

1. Where to run: Local Docker (Recommended — quick, no cloud access needed) / GKE cluster (scale beyond local resources; needs gcloud + kubectl).
2. Scenario (protocol + broadcast mode):
   • Gossipsub, all nodes broadcast (Recommended — the default) → --network-protocol gossipsub --mode all
   • Gossipsub, single broadcaster → --mode one (you MUST also ask for --broadcaster <id>, id < num_nodes, suggest num_nodes - 1; the orchestrator rejects the command at parse without it)
   • Gossipsub, round-robin → --mode rr (nodes take turns every --round-duration-seconds, default 3s)
   • SQMR → --network-protocol sqmr (or reversed-sqmr via Other)
3. Load (nodes × message size @ heartbeat):
   • Default: 3 nodes × 1 KB @ 1000 ms (the binary defaults)
   • Medium: 5 nodes × 4 KB @ 200 ms
   • Heavy: 10 nodes × 16 KB @ 50 ms
4. Duration: 5 min smoke (--timeout-seconds 300) / 15 min (Recommended, 900) / full default (4000 ≈ 66 min) / custom. Always pass --timeout-seconds explicitly — the 66-minute default is a trap for interactive runs.

Then ask ONE short follow-up (multiSelect) — "Change any advanced defaults?":

• None (Recommended) — keep all defaults.
• Network shaping — add ingress --latency (ms) and/or --throughput (KB/s) constraints.
• Buffers & logging — --buffer-size, --verbosity.
• Cluster tuning (only offer for cluster runs) — --node-pool-role, --node-toleration, CPU/memory requests and limits, --production-docker.

Only for the selected categories, ask one more round for the concrete values (offer sensible presets, e.g. latency 50/100/200 ms, with Other for custom). Never ask for values in unselected categories — defaults apply, unless the user already gave values in their prompt.

Flag reference

Step 2: Preflight

Local:

• docker info succeeds.
• Ports 3000, 9090, 8080, and the per-node ranges 2000+id / 10000+id are free (ss -tlnp | grep -E ':(3000|9090|8080)').
• Sanity-check memory: num_nodes × 3g (or --memory-limit) must fit in RAM.

Cluster:

• Verify a live gcloud token, not just a listed account — gcloud auth print-access-token must succeed. An account shown as active by gcloud auth list can still have an expired token that fails reauth headlessly, so the listing check alone is not enough.
• Confirm kubectl config current-context points at the sequencer-dev cluster and a trivial authed call works (kubectl get ns).
• If the token is expired / reauth is required, or the context is wrong, ask the user to run ! gcloud auth login and set the context themselves — never re-auth or switch contexts for them. Do this before cluster start: otherwise a 5–15 min image build runs before the dead token surfaces at the docker push / deploy step.

Parameter guards (validate before starting):

• one mode: --broadcaster is required (the orchestrator rejects the command at parse without it) and its id must be < num_nodes.
• Clock skew (ALL modes on cluster): every received message computes receive_time - send_time, and the node panics if that's negative — i.e. if the sender's clock is even microseconds ahead (handlers.rs). This is mode-independent: all/one/rr all hit it. Single-machine local runs are safe (one clock); on a cluster, nodes crash unless tightly NTP-synced — a real run was killed by 17µs of skew. Warn the user, and treat crashlooping nodes with clock skew detected in their logs as this issue, not a benchmark misconfig.
• rr mode additionally derives turn ownership from each node's wall clock, so skew also corrupts the round schedule (on top of the panic above).
• --message-size-bytes ≥ 36.

Step 3: Start the benchmark

Run in the background (the first run builds a Docker image, which takes minutes):

cargo run --release --bin apollo_network_benchmark_run -- local start \
  --num-nodes <N> --mode <mode> --network-protocol <proto> \
  --message-size-bytes <B> --heartbeat-millis <ms> --timeout-seconds <s>

For --mode one, append --broadcaster <id> (id < num_nodes) — the orchestrator fails at CLI parse without it.

Cluster is identical with cluster start plus any cluster-only flags. Watch the output for the built image tag and successful container/pod startup; on error, surface it to the user and stop — do not retry blindly.

For back-to-back runs, pass --image <ref> to skip the rebuild, using the exact string printed as Image tag: on the previous run (also recorded in the deployment metadata under ~/apollo_network_benchmark_local/ or ~/apollo_network_benchmark_cluster/). The reference differs by mode: local takes the bare local docker tag (broadcast-network-stress-test-node:<timestamp>), while cluster requires the full registry reference (us-central1-docker.pkg.dev/starkware-dev/sequencer/broadcast-network-stress-test-node:<timestamp>) — a bare tag fails the cluster image verification and push.

Step 4: Monitor

• Cluster only: start port-forwarding in the background first:

  cargo run --release --bin apollo_network_benchmark_run -- cluster port-forward
  

  If it exits immediately, the Prometheus/Grafana pods aren't Running yet (no service endpoints) — wait for them and retry: kubectl rollout status statefulset/prometheus -n <ns>.
• Sleepod auto-sleep: on sequencer-dev a sleepod controller sleeps StatefulSets nightly (20:00–05:00 UTC) by default. cluster start opts the run's namespace out with a SleepPolicy, so the dashboards stay up at any run time. If they get napped anyway (scaled to 0, sleepod.io/original-replicas annotation present), the opt-out didn't apply — re-apply the policy, or as a stopgap kubectl scale statefulset/prometheus statefulset/grafana --replicas=1 -n <ns> (note this can be re-slept during the night window).
• Tell the user the dashboards are live: Grafana http://localhost:3000, Prometheus http://localhost:9090.
• Poll the Prometheus HTTP API every few minutes during the run:

  curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=network_connected_peers'
  curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=sum(receive_message_count)'
  

• Health checks (~2 min in): every node should report network_connected_peers = N-1, and receive_message_count should be increasing. If peers never connect or counts stay 0, the run is broken — fetch container/pod logs, report, and offer to stop.
• Report interim snapshots to the user; don't go silent for the whole run.
• Fallback if central Prometheus is unavailable (cluster): the per-node /metrics endpoints vanish the instant the Job completes, so anything not scraped before the timeout is lost. If the central Prometheus/Grafana pods are down, port-forward each node pod and scrape its exporter directly during the run, e.g. kubectl -n <ns> port-forward pod/broadcast-network-stress-test-<id> 2000:2000 && curl -s localhost:2000/metrics. Always capture a final snapshot before the timeout regardless of which path you use.

Step 5: Summarize results

Query Prometheus near the end of the run, before the timeout — once nodes exit, scrape targets disappear and only stale data remains. Instant queries:

# Latency percentiles (seconds). The node exporter renders these as Prometheus SUMMARIES
# (quantile labels), not histograms — there are no `_bucket` series, so histogram_quantile()
# returns empty. Query the quantile labels directly, matching the generated Grafana dashboard.
receive_message_delay_seconds{quantile="0.5"}
receive_message_delay_seconds{quantile="0.95"}
receive_message_delay_seconds{quantile="0.99"}
# Aggregate across nodes:
avg(receive_message_delay_seconds) by (quantile)

# Send-side achieved vs theoretical throughput (bytes/s) — like-for-like, this is the saturation signal
sum(rate(broadcast_message_bytes_sum[5m]))
sum(broadcast_message_theoretical_throughput)

# Receive-side aggregate throughput (bytes/s) — counts every delivered copy, expect ≈ send rate × (N-1)
sum(rate(receive_message_bytes_sum[5m]))

# Delivery accounting
sum(broadcast_message_count)
sum(receive_message_count)
sum(receive_message_pending_count)
sum(network_dropped_broadcast_messages) by (drop_reason)

Report back a short table plus a plain-language verdict:

• The exact command run (so it's reproducible), node count, scenario, duration.
• Latency p50/p95/p99.
• Throughput: send-side achieved vs theoretical, and receive-side aggregate vs expected (send rate × (N-1)).
• Delivery: sent vs received (expected received = sent × (N-1) for broadcast in all/one/rr modes), pending count at end, drops by reason.
• Verdict: healthy (no drops, pending ≈ 0, latency stable), lossy (drops or growing pending), or saturated (send-side achieved throughput plateaus below theoretical, latency climbing).

Step 6: Cleanup

Ask the user before tearing down — they may want to keep dashboards up. Run only the commands for the deployment mode chosen in Step 1; the other mode's commands error when no such deployment exists.

Local:

cargo run --release --bin apollo_network_benchmark_run -- local stop

Verify cleanup with docker ps (no broadcast-network-stress-test containers left).

Cluster:

# Fetch logs first if wanted (saved to /tmp/broadcast-network-stress-test-<id>.logs.txt)
cargo run --release --bin apollo_network_benchmark_run -- cluster logs

cargo run --release --bin apollo_network_benchmark_run -- cluster stop

Kill any background port-forward processes you started.

Gotchas

• 66-minute default timeout: always set --timeout-seconds explicitly for interactive runs.
• Clock-skew panic on cluster (any mode): the receive path panics on a negative receive_time - send_time, so any node whose clock trails a sender's by even microseconds crashes the run — observed killing a --mode all cluster run on 17µs of skew. Local single-machine runs are safe; cluster runs need tightly NTP-synced nodes (or a tolerance change in the node code). rr mode is extra-sensitive since it also keys turn ownership off the wall clock.
• --broadcaster is mandatory for --mode one: the orchestrator flattens UserArgs, whose broadcaster has required_if_eq("mode", "one"), so the command fails at CLI parse without it (the num_nodes - 1 fallback in get_env_var_pairs is unreachable via the CLI). It also errors if the id is ≥ num_nodes.
• --quic does not enable QUIC: it only changes the bootstrap multiaddr format to /udp/<port>/quic-v1; the actual transport is whatever the node binary's NetworkManager builds, which is currently TCP-only (crates/apollo_network/src/network_manager/mod.rs, .with_tcp + TODO: .with_quic()). Until QUIC lands there, passing --quic breaks peer bootstrapping — don't offer it.
• Min message size is 36 bytes (message metadata).
• Port collisions: the local stack claims 3000/9090/8080 plus 2000+id and 10000+id per node — stop other local Grafana/Prometheus stacks first.
• Cluster scheduling failures: pods need the apollo-network-benchmark node pool; if pods stay Pending, check the pool exists and consider --node-toleration or lower --cpu-requests/--memory-requests.
• --production-docker is a slow multi-stage build — only for production-fidelity measurements; the default fast dev build is right for iteration.