Remote Agent HQ Startup & Debug
Troubleshooting guide for starting Agent HQ services on remote machines when they fail or crash.
Trigger Keywords
Use this skill when you see:
- "AMS not responding", "SSE connection lost"
- "port 8041", "port 8037"
- "environment variable", "Missing/invalid AMS port", "Missing/invalid port"
- "vite config", "failed to load config"
- "agent-hq debug", "Agent HQ died"
- "AMS_TMUX_PORT", "AGENT_HQ_UI_PORT", "AGENT_MGMT_PORT"
Quick Reference: Remote Ports
| Service | Port | Purpose |
|---|---|---|
| AMS (Agent Management Server) | 8041 | /api/agent-top/stream SSE endpoint |
| Agent HQ UI | 8037 | Web UI for viewing agents |
| Claude Shim | 8787 | Claude vscode-shim server |
| Codex Shim | 9288 | Codex vscode-shim server |
Problem: AMS Not Responding (SSE Connection Lost)
Symptoms
- UI shows "SSE connection lost"
- "Connecting to SSE..." stuck forever
/api/agent-top/streamreturns connection reset
Diagnosis
# Check if AMS is running on remote
ssh -p <PORT> <HOST> "lsof -nP -iTCP:8041 -sTCP:LISTEN"
Fix: Start AMS
# Start AMS in tmux (requires PATH and env vars)
ssh -p <PORT> <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
cd ~/AgenticProjects/agent-box-v1/services/ams-tmux && \
AMS_TMUX_PORT=8041 ./launchers/run_ams_tmux_server.sh"
If tmux: command not found:
ssh -p <PORT> <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
AMS_TMUX_PORT=8041 \
~/AgenticProjects/agent-box-v1/services/ams-tmux/launchers/run_ams_tmux_server.sh"
Problem: Agent HQ UI Not Responding (port 8037)
Symptoms
http://localhost:28037shows "This site can't be reached"- "ERR_CONNECTION_RESET"
Diagnosis
# Check if UI is running on remote
ssh -p <PORT> <HOST> "lsof -nP -iTCP:8037 -sTCP:LISTEN"
Fix: Start Agent HQ UI
⚠️ CRITICAL: Must source .env file first!
The vite config requires these environment variables:
AMS_TMUX_PORT(orAGENT_MGMT_PORT)AGENT_HQ_UI_PORT(orVITE_PORT)
# Start in tmux with proper env (CORRECT WAY)
ssh -p <PORT> <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
tmux new-session -d -s agent-hq-ui-8037 \
-c ~/AgenticProjects/agent-box-v1/apps/agent-hq-ui \
'bash -lc \"cd ~/AgenticProjects/agent-box-v1 && set -a && source .env && set +a && \
cd apps/agent-hq-ui && npm run dev:web -- --port 8037 --host 127.0.0.1; exec bash\"'"
Common Error:
Error: Missing/invalid AMS port: set AMS_TMUX_PORT (preferred) or AGENT_MGMT_PORT
Error: Missing/invalid port: set AGENT_HQ_UI_PORT (preferred) or VITE_PORT
Cause: .env file not sourced before starting vite.
Fix: Use set -a && source .env && set +a pattern shown above.
Problem: SSH Tunnel Not Forwarding
Symptoms
- Local port not listening
- "Connection refused" when accessing localhost:28xxx
Diagnosis
# Check local tunnel ports
lsof -nP -iTCP:28787 -sTCP:LISTEN
lsof -nP -iTCP:28041 -sTCP:LISTEN
# Check tunnel tmux session
tmux list-sessions | grep tunnel
tmux capture-pane -t ssh-tunnel-to-m2 -p -S -30
Fix: Restart Tunnel
cd ~/swe/vscode-shims && python launchers/launch_ssh_tunnel_to_m2_tmux.py --verbose \
--map 28787:8787,29288:9288,28037:8037,28041:8041
Environment Variables Reference
From ~/AgenticProjects/agent-box-v1/.env:
| Variable | Value | Used By |
|---|---|---|
AMS_TMUX_PORT |
8041 | AMS server, vite config |
AGENT_HQ_UI_PORT |
8037 | Agent HQ UI vite config |
AGENT_MGMT_PORT |
(fallback for AMS_TMUX_PORT) | Legacy |
VITE_PORT |
(fallback for AGENT_HQ_UI_PORT) | Legacy |
Complete Startup Sequence (Remote m2)
# 1. Start AMS (port 8041)
ssh -p 11111 m2@113.161.41.101 "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
cd ~/AgenticProjects/agent-box-v1/services/ams-tmux && \
AMS_TMUX_PORT=8041 ./launchers/run_ams_tmux_server.sh"
# 2. Start Agent HQ UI (port 8037)
ssh -p 11111 m2@113.161.41.101 "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
tmux new-session -d -s agent-hq-ui-8037 \
-c ~/AgenticProjects/agent-box-v1/apps/agent-hq-ui \
'bash -lc \"cd ~/AgenticProjects/agent-box-v1 && set -a && source .env && set +a && \
cd apps/agent-hq-ui && npm run dev:web -- --port 8037 --host 127.0.0.1; exec bash\"'"
# 3. Start SSH tunnel (on LOCAL machine)
cd ~/swe/vscode-shims && python launchers/launch_ssh_tunnel_to_m2_tmux.py --verbose \
--map 28787:8787,29288:9288,28037:8037,28041:8041
# 4. Verify
curl -sS http://127.0.0.1:28041/api/agent-top/stream | head -5 # AMS
curl -sS http://127.0.0.1:28037/ | head -5 # Agent HQ UI
Tmux Session Names
| Session | Service |
|---|---|
ams_tmux |
AMS server (port 8041) |
agent-hq-ui-8037 |
Agent HQ UI (port 8037) |
ssh-tunnel-to-m2 |
SSH tunnel (local) |
Related Skills
- agentic-ecosystem-remote-deployment - Full deployment guide
- config-transparency-centralized - .env philosophy