Local Source

Skill: Local Source Setup

Objective

Build Danube from the source repository and run brokers locally. This setup is for users who are developing on the Danube codebase and want to test their local changes.

Prerequisites

Run the prerequisites check before setup:

./scripts/check_prereqs.sh source

This verifies Rust toolchain (cargo, rustc), make, Danube source repo location, and port availability.

Note: The AI must also ask the user for the Danube source repository path. Do not assume or hardcode any path.

How to Run

Once prerequisites are confirmed, run the setup script with the user-provided repo path:

# Build and start a 3-broker cluster from source
./scripts/setup_local_source.sh /path/to/danube 3

# Cleanup
./scripts/cleanup.sh source

The script is at scripts/setup_local_source.sh — read it for the full implementation details.

Key Concepts

The Makefile Drives Everything

The Danube source repo has a Makefile that handles building and running brokers. The setup script calls make brokers — it does not build or start brokers manually.

Logs are saved to temp/broker_<port>.log in the Danube repo directory (not in $TEST_RUN).

This Setup Is Cluster-Only

The Makefile always runs brokers in cluster mode (with config file + seed nodes). For standalone mode, use setups/local-binary/ instead.

Config Options

• Default: The Makefile uses ./config/danube_broker.yml from the Danube repo
• Custom: Pass CONFIG_FILE=<path> to make (e.g., make brokers CONFIG_FILE=path/to/config.yml)
• Test-specific: The repo includes configs in config/for_tests/ (rebalance, shared_fs, write_buffer)

Binaries Are Built Locally

make brokers only builds danube-broker. The setup script builds danube-admin separately for verification (cargo build --release --package danube-admin). Binaries are at $DANUBE_REPO/target/release/.

Available Makefile Targets

Makefile Variables

Verification

The setup script (scripts/setup_local_source.sh) runs these checks automatically. The expected output is documented here so the AI can confirm the setup is healthy.

./target/release/danube-admin brokers list

All brokers must show status active. One broker has role Cluster_Leader, the rest are Cluster_Follower.

BROKER ID       STATUS   ADDRESS              ROLE              ADMIN ADDR
---------------------------------------------------------------------------
5804156356...   active   http://0.0.0.0:6650  Cluster_Leader    http://0.0.0.0:50051
9393761688...   active   http://0.0.0.0:6651  Cluster_Follower  http://0.0.0.0:50052
1293191161...   active   http://0.0.0.0:6652  Cluster_Follower  http://0.0.0.0:50053

./target/release/danube-admin cluster status

The Leader field must show a valid node ID (not none). All broker node IDs should appear in the Voters list.

Raft Cluster Status:
  Self Node ID:  5804156356532636512
  Raft Address:  0.0.0.0:7650
  Leader:        5804156356532636512
  Term:          1
  Last Applied:  18
  Voters:        [5804156356532636512, 9393761688591103413, 12931911617355319510]

Fail indicators:

• Any broker with status other than active
• Leader: none in cluster status (no leader elected)
• Fewer voters than expected brokers
• ERROR, PANIC, or FATAL in broker logs: grep -i "ERROR\|PANIC\|FATAL" temp/broker_*.log

Troubleshooting

• Cargo build fails: Ensure Rust toolchain is up to date: rustup update. Check the repo's rust-toolchain.toml or Cargo.toml for the required Rust version.

• Broker crashes on start: Check the log file: tail -50 temp/broker_6650.log. Common causes: port conflict, corrupted Raft data.

• Raft cluster won't form: Delete stale data: make data-clean, then restart: make brokers.

• Build is slow: First build compiles all dependencies (~5-10 minutes). Subsequent builds are incremental and much faster.

• Can't find the binary: After make brokers, binaries are at $DANUBE_REPO/target/release/danube-broker (and danube-cli, danube-admin).