tomevault-io Node Operations
Operational guide for running, monitoring, and troubleshooting the Midnight node. For configuration options, see midnight-node:node-configuration. For architecture internals, see midnight-node:node-architecture.
Running Modes
Validator Node
Produces blocks and participates in consensus. Requires all four validator keys.
midnight-node --chain preview \
--validator \
--name "my-validator"
Validator nodes must have:
- AURA, GRANDPA, CROSS_CHAIN, and BEEFY keys configured (see
node-configuration) - Sufficient peers for consensus participation
- Stable network connectivity and uptime
Full Node
Syncs and validates the chain without producing blocks. Serves RPC requests and provides network capacity.
midnight-node --chain preview \
--name "my-full-node" \
--rpc-external --rpc-cors all
Security Warning:
--rpc-external --rpc-cors allexposes the RPC interface to all network interfaces without restriction. Use only for local development. In production, use a reverse proxy with TLS and restrict CORS origins.
Archive Node
Retains all historical state. Required for block explorers, indexers, and historical queries.
midnight-node --chain preview \
--state-pruning archive \
--blocks-pruning archive \
--name "my-archive-node"
Development Mode
Single-node ephemeral chain for local development and testing.
midnight-node --dev --tmp
Development mode automatically:
- Creates a single-validator chain
- Uses pre-funded development accounts
- Enables all RPC methods
- Uses ephemeral storage (deleted on shutdown with
--tmp)
Hardware Requirements
Minimum recommended specifications by node type:
| Node Type | CPU | RAM | Storage |
|---|---|---|---|
| Full node | 4 cores | 8 GB | 100 GB SSD |
| Archive node | 4 cores | 16 GB | 500 GB SSD |
| Validator | 8 cores | 16 GB | 200 GB SSD (NVMe recommended) |
These are estimates for a Substrate-based chain and may increase as chain state grows. Monitor disk usage and plan capacity accordingly.
Docker Deployment
The Midnight node is distributed as a Docker image via midnight-node-docker.
Replace
<version>with the latest release tag. Check the Midnight release notes for the current version.
# Pull the node image
docker pull midnightntwrk/midnight-node:<version>
# Run a full node
docker run -d --name midnight-node \
-p 9944:9944 \
-p 30333:30333 \
-p 9615:9615 \
-v midnight-data:/data \
midnightntwrk/midnight-node:<version> \
midnight-node --chain preview \
--rpc-external --rpc-cors all \
--prometheus-external
Security Warning:
--rpc-external --rpc-cors allexposes the RPC interface to all network interfaces without restriction. Use only for local development. In production, use a reverse proxy with TLS and restrict CORS origins.
Port Mapping
| Port | Protocol | Purpose |
|---|---|---|
| 9944 | WebSocket | JSON-RPC API |
| 30333 | TCP | P2P networking |
| 9615 | HTTP | Prometheus metrics |
Volume Mounts
| Path | Purpose |
|---|---|
/data |
Chain database and ledger storage |
/keys |
Validator key files (validator mode only) |
Monitoring
Prometheus Metrics
The node exposes Prometheus metrics on port 9615 by default.
# Enable external Prometheus endpoint
midnight-node --chain preview --prometheus-external
# Scrape metrics
curl http://localhost:9615/metrics
Key metrics to monitor:
| Metric | Description |
|---|---|
substrate_block_height |
Current best and finalized block heights |
substrate_number_leaves |
Number of leaves in the state trie |
substrate_peers_count |
Number of connected peers |
substrate_ready_transactions_number |
Transactions in the ready queue |
substrate_sync_peers |
Number of sync peers |
substrate_block_verification_time |
Block import verification time |
Push Endpoint
The node supports pushing metrics to a remote Prometheus Pushgateway via the PROMETHEUS_PUSH_ENDPOINT environment variable.
PROMETHEUS_PUSH_ENDPOINT=http://pushgateway:9091 midnight-node --chain preview
Memory Monitoring
The node includes a configurable memory monitoring system. When memory usage exceeds the configured threshold, the node initiates a graceful shutdown to prevent out-of-memory crashes.
| Parameter | Description |
|---|---|
memory_threshold |
Percentage of system memory that triggers shutdown (e.g., 90) |
The node logs memory usage periodically and emits a warning before initiating shutdown, allowing the process manager (systemd, Docker, Kubernetes) to restart it cleanly.
P2P Networking
Bootnodes
Bootnodes are the initial peers the node contacts to discover the network. Each network preset includes default bootnodes.
# Add custom bootnodes
midnight-node --chain preview \
--bootnodes /dns4/bootnode.example.com/tcp/30333/p2p/12D3KooW...
Node Keys
Each node has a unique libp2p identity derived from a node key. The key is auto-generated on first run and stored in the database directory.
# Generate a node key manually
midnight-node key generate-node-key --file /keys/node.key
# Use a specific node key
midnight-node --chain preview --node-key-file /keys/node.key
P2P Port
The default P2P port is 30333. Ensure this port is open for inbound connections if the node should be reachable by peers.
midnight-node --chain preview --port 30333 --listen-addr /ip4/0.0.0.0/tcp/30333
Troubleshooting
Blocks Not Finalizing
| Symptom | Possible Cause | Resolution |
|---|---|---|
| Best block increases but finalized block is stuck | GRANDPA consensus stalled | Check peer count (system_health RPC); ensure validator keys are correct |
| Finalized block lags far behind best block | Network partition or insufficient validators | Verify bootnodes are reachable; check GRANDPA round state via grandpa_roundState |
| No blocks produced at all | AURA authority not in active set | Verify AURA key is registered; check epoch configuration |
Mainchain Follower Errors
| Symptom | Possible Cause | Resolution |
|---|---|---|
MainchainFollower connection failed |
PostgreSQL db-sync unreachable | Verify db-sync is running and PostgreSQL connection string is correct |
MainchainFollower sync behind |
db-sync is still syncing Cardano chain | Wait for db-sync to reach tip; check db-sync logs |
| Partner chain data stale | db-sync stopped or restarting | Restart db-sync; verify Cardano node connectivity |
IPC Errors
| Symptom | Possible Cause | Resolution |
|---|---|---|
| IPC socket connection refused | Node process crashed or socket file stale | Restart the node; remove stale socket files from data directory |
| IPC timeout | Node under heavy load | Increase system resources; check CPU and memory usage |
State Sync Issues
| Symptom | Possible Cause | Resolution |
|---|---|---|
| Sync stalls at a specific block | Corrupted state or database | Clear chain data and resync: remove database directory, restart node |
State database error |
Disk full or I/O error | Check disk space; verify storage health |
| Very slow sync | Insufficient peers or slow network | Add more bootnodes; verify network connectivity; increase --db-cache |
Diagnostic Commands
# Check node health
curl -s -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"system_health","params":[]}' \
http://localhost:9944
# Check sync state
curl -s -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"system_syncState","params":[]}' \
http://localhost:9944
# Check connected peers
curl -s -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"system_peers","params":[]}' \
http://localhost:9944
# Check GRANDPA round state
curl -s -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"grandpa_roundState","params":[]}' \
http://localhost:9944
Cross-References
midnight-node:node-configuration— Detailed configuration flags and environment variablesmidnight-node:node-rpc-api— RPC methods used for monitoring and diagnosticsmidnight-tooling:devnet— Local development stack that manages node lifecycle automatically
Source: devrelaicom/midnight-expert — distributed by TomeVault.