Dagster Remote

Remote Dagster Server Manager

Manage the Dagster pipeline server running on dns-analy4 for the scan-analysis-prod project.

Environment

Operations

Check Status

# Check if tmux session exists and services are running
ssh dns-analy4 "tmux has-session -t dagster 2>&1 && echo 'tmux: running' || echo 'tmux: not running'"

# View recent log output
ssh dns-analy4 "tmux capture-pane -t dagster -p | tail -20"

# Check processes directly
ssh dns-analy4 "pgrep -fa 'dagster' | grep -v grep"

Start Server

# Start in tmux (recommended)
ssh dns-analy4 "cd /home/zhaoliang/Projects/scan-analysis-prod && tmux new-session -d -s dagster 'bash scripts/dagster-remote.sh'"

Stop Server

# Send Ctrl-C to tmux session (graceful shutdown)
ssh dns-analy4 "tmux send-keys -t dagster C-c"

# Or kill the tmux session
ssh dns-analy4 "tmux kill-session -t dagster"

# Force kill if needed
ssh dns-analy4 "pkill -f 'dagster-webserver|dagster-daemon'"

Restart Server

# Kill existing session, then start fresh
ssh dns-analy4 "tmux kill-session -t dagster 2>/dev/null; cd /home/zhaoliang/Projects/scan-analysis-prod && tmux new-session -d -s dagster 'bash scripts/dagster-remote.sh'"

Sync Code (Local → Remote)

Run the setup script from local machine to sync code and dependencies:

bash scripts/setup-remote.sh

This script:

1. Installs/updates uv on dns-analy4
2. Rsyncs project files (excludes .venv, data, results, .git)
3. Runs uv sync for dependencies
4. Copies .env.remote and symlinks as .env
5. Copies GeoIP database if missing
6. Initializes DAGSTER_HOME

After syncing, restart the server to pick up code changes.

Sync Data (Remote → Local)

Sync partitioned and non-partitioned parquet asset data from dns-analy4 to local. Run both in parallel for speed.

# Sync partitioned parquet data
rsync -avz --progress dns-analy4:/home/zhaoliang/Projects/scan-analysis-prod/data/partitions/ /Users/zhaoliang/Projects/scan-analysis-prod/data/partitions/

# Sync non-partitioned asset parquet files
rsync -avz --progress dns-analy4:/home/zhaoliang/Projects/scan-analysis-prod/data/assets/ /Users/zhaoliang/Projects/scan-analysis-prod/data/assets/

After syncing data, rebuild DuckDB views to reflect new/updated parquet files:

cd /Users/zhaoliang/Projects/scan-analysis-prod && uv run python scripts/sync_parquet_to_duckdb.py

Sync Results (Remote → Local)

Sync generated figures (PDF/PNG) and LaTeX tables from dns-analy4 to local.

rsync -avz --progress dns-analy4:/home/zhaoliang/Projects/scan-analysis-prod/result/ /Users/zhaoliang/Projects/scan-analysis-prod/result/

Sync All (Remote → Local)

To sync everything (data + results) in one go, run all three rsync commands in parallel:

rsync -avz --progress dns-analy4:/home/zhaoliang/Projects/scan-analysis-prod/data/partitions/ /Users/zhaoliang/Projects/scan-analysis-prod/data/partitions/ &
rsync -avz --progress dns-analy4:/home/zhaoliang/Projects/scan-analysis-prod/data/assets/ /Users/zhaoliang/Projects/scan-analysis-prod/data/assets/ &
rsync -avz --progress dns-analy4:/home/zhaoliang/Projects/scan-analysis-prod/result/ /Users/zhaoliang/Projects/scan-analysis-prod/result/ &
wait
echo "All syncs complete"

Access Dagster UI from Local

# SSH tunnel for web UI
ssh -L 3000:localhost:3000 dns-analy4

# Then open http://localhost:3000

View Logs

# Live tmux output
ssh dns-analy4 "tmux capture-pane -t dagster -p -S -100"

# Check active/queued runs
ssh dns-analy4 "cd /home/zhaoliang/Projects/scan-analysis-prod && ~/.local/bin/uv run dagster run list -m awsntpdagster.definitions --limit 5"

Configuration

Dagster Config (config/prod.yaml → DAGSTER_HOME/dagster.yaml)

Key settings:

• run_queue.max_concurrent_runs: 10
• concurrency.default_op_concurrency_limit: 10
• run_retries.enabled: true (max 2 retries)

Environment Variables (.env.remote)

• ENVIRONMENT=remote
• ARKIME_PORT=8042
• DAGSTER_WEBSERVER_PORT=3000
• SKIP_EXTRACTION=0 (queries Arkime directly)
• GEOIP_ASN_DB_PATH points to local copy on dns-analy4

Troubleshooting

tmux "no server running"

The tmux server isn't running. Just start a new session:

ssh dns-analy4 "cd /home/zhaoliang/Projects/scan-analysis-prod && tmux new-session -d -s dagster 'bash scripts/dagster-remote.sh'"

"uv not found" in dagster-remote.sh

The script adds ~/.local/bin to PATH. If uv was installed elsewhere, check:

ssh dns-analy4 "which uv"

Database Lock Error

If daemon and webserver both try to access SQLite simultaneously, the startup script has a 5-second delay between daemon and webserver starts. If it persists, restart.

Redis Unavailable Warning

Expected — Redis is not installed on dns-analy4. The pipeline runs without caching (direct Arkime requests each time). This is a warning, not an error.

Port Already in Use

ssh dns-analy4 "lsof -i :3000"
# Kill stale process if needed
ssh dns-analy4 "pkill -f 'dagster-webserver'"