# Agentic Ecosystem Deployment

> Bootstrap the complete agentic ecosystem (Claude Code, Codex, vscode-shims, agent-box) on a remote machine. Use when user asks to install ecosystem, setup remote agent, bootstrap agent box, deploy to remote, or install claude/codex on another machine.

- Skill: `tankygranny05/agentic-ecosystem-deployment` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add tankygranny05/agentic-ecosystem-deployment`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tankygranny05/agentic-ecosystem-deployment/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: tankygranny05 (https://skillmd.com/u/tankygranny05)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/tankygranny05/agentic-ecosystem-deployment

---

<!-- [Created by Claude: 67f7b402-050c-4754-bdad-8c5e777eddd2] -->
<!-- [Edited by Claude: 1446bf2e-3669-4050-a5ba-48be09357f30 2026-01-25] -->
<!-- [Edited by Claude: de426bd4-2dc6-4189-b2f9-a865d757e726 2026-01-25] -->

# Agentic Ecosystem Remote Bootstrap

Complete guide for deploying the agentic development ecosystem to a remote machine, including Claude Code, Codex CLI, vscode-shims, and supporting infrastructure.

## Quick Start Checklist

```bash
# 1. Verify SSH access
ssh -p <PORT> <USER>@<HOST> "uname -a"

# 2. Install prerequisites (Node.js, Cargo)
ssh -p <PORT> <HOST> "/opt/homebrew/bin/brew install node"
# Cargo usually pre-installed, verify with: cargo --version

# 3. Deploy Claude Code + Codex (see detailed steps below)

# 4. Create symlinks
ssh -p <PORT> <HOST> "mkdir -p ~/symlinks && \
  ln -sf ~/swe/claude-code-X.Y.Z/cli.js ~/symlinks/claude && \
  ln -sf ~/swe/codex.X.Y.Z/codex-rs/target/release/codex ~/symlinks/codex"

# 5. Deploy auth (see authentication section)

# 6. Smoke test
ssh -p <PORT> <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
  ~/symlinks/claude -p 'What is 2 + 2?'"
ssh -p <PORT> <HOST> "~/symlinks/codex exec --skip-git-repo-check 'What is 2 + 2?'"
```

---

## Critical Knowledge (Lessons Learned)

### ⚠️ IMPORTANT: Check Remote System Info First

Before any deployment, gather system details:

```bash
# Check OS, architecture, memory
ssh -p <PORT> <HOST> "uname -a && sysctl hw.memsize 2>/dev/null | awk '{print \$2/1024/1024/1024 \" GB\"}'"

# Check for Node.js
ssh -p <PORT> <HOST> "which node && node --version || echo 'Node.js NOT installed'"

# Check for Cargo
ssh -p <PORT> <HOST> "which cargo && cargo --version || echo 'Cargo NOT installed'"

# Check for Homebrew
ssh -p <PORT> <HOST> "which brew || ls /opt/homebrew/bin/brew"
```

### ⚠️ IMPORTANT: Trace Symlinks to Find Correct Versions

Always check `~/symlinks/` on LOCAL machine to determine which versions to deploy:

```bash
# On LOCAL machine - check what versions to deploy
ls -la ~/symlinks/claude ~/symlinks/codex
# Example output:
# claude -> /Users/sotola/swe/claude-code-2.1.15/cli.js
# codex -> /Users/sotola/swe/codex.0.88.0/codex-rs/target/release/codex
```

**NEVER assume version numbers** - always trace the symlinks!

### ⚠️ IMPORTANT: Exclude Large Files When Transferring

When cloning projects for transfer, ALWAYS exclude:

| Exclude | Reason | Typical Size |
|---------|--------|--------------|
| `node_modules/` | Reinstall on target | 50-500 MB |
| `.git/` | Not needed for deployment | 10-100 MB |
| `target/` (Rust) | Rebuild on target | 500+ MB |
| `.electron-dev/` | Electron binaries | 150+ MB |
| `__pycache__/` | Python cache | Small but unnecessary |

**rsync exclude pattern:**
```bash
rsync -av --exclude='.git' --exclude='node_modules' --exclude='target' \
  --exclude='.electron-dev' --exclude='__pycache__' --exclude='*.pyc' \
  <source>/ /tmp/<project>-clean/
```

### ⚠️ IMPORTANT: PATH Issues with SSH Non-Interactive Shell

SSH commands run in non-interactive shells that DON'T load `~/.zshrc`. Always export PATH:

```bash
# WRONG - node not found
ssh <HOST> "node --version"

# CORRECT - explicitly set PATH
ssh <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && node --version"
```

**CRITICAL for shim servers:** When starting vscode-shims remotely, the Python process inherits the limited SSH PATH. If `/opt/homebrew/bin` is not included, the shim will fail to spawn Claude CLI with error: `[Errno 2] No such file or directory: 'node'`. This error appears in the webview UI, not in SSH output.

**Always start shims with:**
```bash
ssh <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
  cd ~/swe/vscode-shims && \
  nohup ~/anaconda3/bin/python src/claude/server.py > /tmp/claude-shim.log 2>&1 &"
```

### ⚠️ IMPORTANT: python vs python3 on macOS

macOS has `python3` but often no `python` alias:

```bash
# WRONG - command not found
ssh <HOST> "python script.py"

# CORRECT - use python3
ssh <HOST> "python3 script.py"
```

**Fix launcher scripts** that use `python` to use `python3` instead.

### ⚠️ IMPORTANT: npm install Needs AUTHORIZED=1 for Claude Code

Claude Code package.json has a protection script that blocks direct installs:

```bash
# WRONG - fails with "Direct publishing is not allowed"
npm install

# CORRECT - set AUTHORIZED flag
AUTHORIZED=1 npm install
```

### ⚠️ IMPORTANT: Hardcoded Paths in server.py

The vscode-shims server.py may have hardcoded fallback paths:

```python
# server.py line ~940 - PROBLEMATIC!
cli_path = os.environ.get("CLAUDE_CODE_CLI") or \
  os.path.expanduser("~/swe/claude-code-2.1.12/cli.js")  # <-- HARDCODED!
```

**Fix:** Always set `CLAUDE_CODE_CLI` in `.env` or use `--cli` flag:

```bash
# In .env
CLAUDE_CODE_CLI=~/symlinks/claude

# Or via command line
python3 server.py --cli ~/symlinks/claude
```

### ⚠️ IMPORTANT: SHIM_DEFAULT_CWD Directory Must Exist

The shim server requires the working directory to exist:

```bash
# Error: Failed to spawn CLI: [Errno 2] No such file or directory: '/Users/sotola/agent-home'
```

**Fix:** Create the directory or sync from local:

```bash
# Simple fix
ssh <HOST> "mkdir -p ~/agent-home"

# Better - sync structure with config files
rsync -av --include='*/' --include='*.md' \
  --exclude='*' -e "ssh -p <PORT>" ~/agent-home/ <HOST>:~/agent-home/
```

### ⚠️ IMPORTANT: Network Binding - NEVER Use 0.0.0.0

When configuring `SHIM_HOST` for network access:

```bash
# DANGEROUS - exposes to ALL networks including internet!
SHIM_HOST=0.0.0.0

# CORRECT - bind to specific LAN interface
SHIM_HOST=192.168.1.9  # Your machine's LAN IP
```

**Security levels:**
- `127.0.0.1` = localhost only (most secure, requires SSH tunnel)
- `192.168.1.X` = LAN interface only (secure for trusted network)
- ~~`0.0.0.0`~~ = **NEVER USE** - exposes to all networks

### ⚠️ IMPORTANT: Codex Shim Needs Python Dependencies

The Codex shim server requires `requests` module:

```bash
# Error: ModuleNotFoundError: No module named 'requests'

# Fix
pip3 install requests
```

---

## Component Deployment

### 1. Claude Code Deployment

**On LOCAL machine:**

```bash
# 1. Check which version to deploy (trace symlink)
readlink ~/symlinks/claude
# Output: /Users/sotola/swe/claude-code-2.1.15/cli.js

# 2. Clone to /tmp excluding node_modules and .git
cp -r ~/swe/claude-code-2.1.15 /tmp/claude-code-2.1.15
cd /tmp/claude-code-2.1.15
rm -rf node_modules .git soto_docs soto_reqs ai

# 3. Check size (should be ~70-80 MB without node_modules)
du -sh /tmp/claude-code-2.1.15

# 4. Zip for transfer
cd /tmp && zip -r claude-code-2.1.15.zip claude-code-2.1.15

# 5. Transfer to remote
scp -P <PORT> /tmp/claude-code-2.1.15.zip <HOST>:/tmp/
```

**On REMOTE machine:**

```bash
# 1. Extract
ssh -p <PORT> <HOST> "cd /tmp && unzip -q claude-code-2.1.15.zip && \
  mkdir -p ~/swe && mv claude-code-2.1.15 ~/swe/"

# 2. Install dependencies (note AUTHORIZED=1)
ssh -p <PORT> <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
  cd ~/swe/claude-code-2.1.15 && AUTHORIZED=1 npm install"

# 3. Create symlink
ssh -p <PORT> <HOST> "mkdir -p ~/symlinks && \
  ln -sf ~/swe/claude-code-2.1.15/cli.js ~/symlinks/claude && \
  ln -sf ~/swe/claude-code-2.1.15 ~/symlinks/claude-code-2.1.15"

# 4. Verify
ssh -p <PORT> <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
  ~/symlinks/claude --version"
# Expected: 2.1.15 (Claude Code)
```

### 2. Codex Deployment

**On LOCAL machine:**

```bash
# 1. Check which version to deploy (trace symlink)
readlink ~/symlinks/codex
# Output: /Users/sotola/swe/codex.0.88.0/codex-rs/target/release/codex

# 2. Clone excluding target, node_modules, .git
cp -r ~/swe/codex.0.88.0 /tmp/codex.0.88.0
cd /tmp/codex.0.88.0
rm -rf codex-rs/target node_modules .git soto_docs

# 3. Check size (should be ~15-20 MB)
du -sh /tmp/codex.0.88.0

# 4. Zip for transfer
cd /tmp && zip -r codex.0.88.0.zip codex.0.88.0

# 5. Transfer to remote
scp -P <PORT> /tmp/codex.0.88.0.zip <HOST>:/tmp/
```

**On REMOTE machine:**

```bash
# 1. Extract
ssh -p <PORT> <HOST> "cd /tmp && unzip -q codex.0.88.0.zip && \
  mkdir -p ~/swe && mv codex.0.88.0 ~/swe/"

# 2. Build with cargo (takes several minutes)
ssh -p <PORT> <HOST> "cd ~/swe/codex.0.88.0/codex-rs && cargo build --release"

# 3. Create symlink
ssh -p <PORT> <HOST> "mkdir -p ~/symlinks && \
  ln -sf ~/swe/codex.0.88.0/codex-rs/target/release/codex ~/symlinks/codex"

# 4. Verify
ssh -p <PORT> <HOST> "~/symlinks/codex --version"
# Expected: codex-cli 0.88.0
```

### 3. vscode-shims Deployment

**On LOCAL machine:**

```bash
# 1. Clone excluding unnecessary files
rsync -av --exclude='.git' --exclude='node_modules' --exclude='.playwright-mcp' \
  --exclude='__pycache__' --exclude='*.pyc' \
  ~/swe/vscode-shims/ /tmp/vscode-shims-clean/

# 2. Check size
du -sh /tmp/vscode-shims-clean  # Should be ~70-80 MB

# 3. Transfer to remote
rsync -avz --progress -e "ssh -p <PORT>" \
  /tmp/vscode-shims-clean/ <HOST>:~/swe/vscode-shims/

# 4. Clean up local temp
rm -rf /tmp/vscode-shims-clean
```

**On REMOTE machine - Configure .env:**

```bash
ssh -p <PORT> <HOST> "cat >> ~/swe/vscode-shims/.env << 'EOF'

# CLI Paths (use symlinks for version flexibility)
CLAUDE_CODE_CLI=~/symlinks/claude
CODEX_CLI=~/symlinks/codex

# Network Access Configuration
# 127.0.0.1 = localhost only (requires SSH tunnel)
# 192.168.1.X = LAN access (replace with your machine's LAN IP)
SHIM_HOST=127.0.0.1
CODEX_SHIM_HOST=127.0.0.1
EOF"
```

**Fix launcher script (python -> python3):**

```bash
ssh -p <PORT> <HOST> "sed -i '' 's/python src/python3 src/g' \
  ~/swe/vscode-shims/launchers/launch_server.sh"
```

### 4. agent-home Directory

```bash
# Sync directory structure with CLAUDE.md (no large data files)
rsync -av --include='*/' --include='CLAUDE.md' --include='AGENTS.md' \
  --include='.mcp.json' --exclude='*' \
  -e "ssh -p <PORT>" ~/agent-home/ <HOST>:~/agent-home/
```

### 5. Telemetry Projects (Optional)

```bash
# Transfer each project
for project in codex_sqlite claude_sqlite agent_dump; do
  rsync -avz --progress -e "ssh -p <PORT>" \
    ~/swe/telemetry_projects/$project <HOST>:~/swe/telemetry_projects/
done
```

### 6. Agent Box (Optional)

```bash
# Clone excluding large files
rsync -av --exclude='.git' --exclude='node_modules' --exclude='.electron-dev' \
  --exclude='__pycache__' --exclude='*.pyc' \
  ~/AgenticProjects/agent-box-v1/ /tmp/agent-box-v1-clean/

# Transfer
rsync -avz --progress -e "ssh -p <PORT>" \
  /tmp/agent-box-v1-clean/ <HOST>:~/AgenticProjects/agent-box-v1/

# Clean up
rm -rf /tmp/agent-box-v1-clean
```

---

## Authentication Deployment

### Claude Code Auth

See skill: **claude-usage-meter** for full OAuth refresh workflow.

**Automated upload to multiple remotes (recommended):**

```bash
# Upload to all configured remotes (cm3u, cm2, etc.)
python ~/.claude/skills/claude-usage-meter/scripts/upload_credentials_to_remotes.py

# Upload to specific remotes
python ~/.claude/skills/claude-usage-meter/scripts/upload_credentials_to_remotes.py cm3u cm2
```

The upload script:
- Reads SSH aliases from `~/.zshrc`
- Uploads `~/.claude/.credentials.json` to remote `~/.claude/`
- Sets correct permissions (600)
- Tests each remote after upload
- Shows success/failure summary

**Manual upload (if script unavailable):**

```bash
# Copy credentials to remote
scp -P <PORT> ~/.claude/.credentials.json <HOST>:~/.claude/.credentials.json

# Via SSH alias with pipe
cat ~/.claude/.credentials.json | cm3u 'cat > ~/.claude/.credentials.json && chmod 600 ~/.claude/.credentials.json'
```

### Codex Auth

See skill: **codex-remote-auth** for details.

```bash
# Simple copy
scp -P <PORT> ~/.codex/auth.json <HOST>:~/.codex/auth.json
```

---

## Smoke Test Commands

### Test Claude Code CLI

```bash
ssh -p <PORT> <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
  ~/symlinks/claude -p 'What is 2 + 2?'"

# Expected output: "2 + 2 = 4" or similar
# Auth error is OK - confirms binary works
```

### Test Codex CLI

```bash
ssh -p <PORT> <HOST> \
  "~/symlinks/codex exec --skip-git-repo-check 'What is 2 + 2?'"

# Expected: Shows "codex-cli X.Y.Z" header, then response or auth error
# Auth error is OK - confirms binary works
```

### Check Versions

```bash
ssh -p <PORT> <HOST> \
  "~/symlinks/codex --version && echo '---' && \
   export PATH=\"/opt/homebrew/bin:\$PATH\" && ~/symlinks/claude --version"

# Expected:
# codex-cli 0.88.0
# ---
# 2.1.15 (Claude Code)
```

### Test Shim Server

```bash
# Start server (CRITICAL: export PATH for node)
ssh -p <PORT> <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
  cd ~/swe/vscode-shims && \
  nohup ~/anaconda3/bin/python src/claude/server.py > /tmp/claude-shim.log 2>&1 &"

# Wait and check
sleep 3
ssh -p <PORT> <HOST> "lsof -i :8787"

# Curl test
ssh -p <PORT> <HOST> "curl -s http://localhost:8787/ | head -20"
# Expected: HTML with "Claude Webview Shim"
```

**⚠️ CRITICAL:** Always export PATH="/opt/homebrew/bin:$PATH" before starting shims, otherwise node will not be found when spawning CLI.

### SSH Tunnel for Local Access

```bash
# Create tunnel (run on LOCAL machine)
ssh -p <PORT> -f -N -L 18787:localhost:8787 <USER>@<HOST>

# Access at http://localhost:18787
```

---

## Port Mapping for SSH Tunnels

<!-- [Added by Claude: 3ae26ab6-64d2-4932-8ed8-4ca6f4313658 2026-01-25] -->

### ⚠️ CRITICAL: Reserve Local Ports for Local Services

**NEVER tunnel remote services to ports already used locally.** For example:
- Port `8787` is used by **local** Claude vscode-shim
- Port `9288` is used by **local** Codex vscode-shim
- Port `8037` is used by **local** Agent HQ

If you tunnel remote:8787 to local:8787, you'll kill your local service!

### Port Mapping Convention

Use `+20000` offset for remote tunnels:

| Service | Local Port | Remote Tunnel Port |
|---------|------------|-------------------|
| Claude shim | 8787 | 28787 |
| Codex shim | 9288 | 29288 |
| Agent HQ | 8037 | 28037 |

### Using the Tunnel Launcher Script

The `launch_ssh_tunnel_to_m2_tmux.py` script in `~/swe/vscode-shims/launchers/` supports explicit port mapping:

```bash
# Start tunnel with proper port mapping (avoids local port conflicts)
cd ~/swe/vscode-shims && \
  python launchers/launch_ssh_tunnel_to_m2_tmux.py --verbose \
  --map 28787:8787,29288:9288,28037:8037
```

This creates:
- `localhost:28787` → `remote:8787` (Claude shim)
- `localhost:29288` → `remote:9288` (Codex shim)
- `localhost:28037` → `remote:8037` (Agent HQ)

### Verifying Tunnel is Working

```bash
# Check tunnel ports are listening locally
lsof -nP -iTCP:28787 -sTCP:LISTEN
lsof -nP -iTCP:29288 -sTCP:LISTEN
lsof -nP -iTCP:28037 -sTCP:LISTEN

# Curl test through tunnel
curl -sS -m 3 http://127.0.0.1:28787/ | head -5
```

### Tmux Session Management

The tunnel runs in tmux session `ssh-tunnel-to-m2`:

```bash
# Attach to see tunnel status/logs
tmux attach -t ssh-tunnel-to-m2

# List sessions
tmux list-sessions | grep tunnel

# Restart tunnel (re-run the launcher)
python launchers/launch_ssh_tunnel_to_m2_tmux.py --verbose --map 28787:8787,29288:9288,28037:8037
```

---

## Post-Deployment Smoke Testing

<!-- [Added by Claude: 3ae26ab6-64d2-4932-8ed8-4ca6f4313658 2026-01-25] -->

### 1. Start Local Services First

Before testing tunnels, ensure local services are running:

```bash
# Start local vscode-shim (claims port 8787)
cd ~/swe/vscode-shims && python launchers/launch_server_tmux.py

# Verify local service
curl -sS http://127.0.0.1:8787/ | head -3
```

### 2. Start SSH Tunnel with Port Mapping

```bash
cd ~/swe/vscode-shims && \
  python launchers/launch_ssh_tunnel_to_m2_tmux.py --verbose \
  --map 28787:8787,29288:9288,28037:8037
```

### 3. Verify Both Local and Remote Access

```bash
# Local service (direct)
curl -sS http://127.0.0.1:8787/ | head -3
# Expected: HTML with "Claude Webview Shim"

# Remote service (via tunnel)
curl -sS http://127.0.0.1:28787/ | head -3
# Expected: HTML with "Claude Webview Shim" (from remote)
```

### 4. Browser Access URLs

| Service | Local URL | Remote (via tunnel) |
|---------|-----------|---------------------|
| Claude shim | http://localhost:8787 | http://localhost:28787 |
| Codex shim | http://localhost:9288 | http://localhost:29288 |
| Agent HQ | http://localhost:8037 | http://localhost:28037 |

### 5. Troubleshooting Tunnel Issues

**Port not listening after starting tunnel:**
```bash
# Check tmux session for errors
tmux capture-pane -t ssh-tunnel-to-m2 -p -S -30 | tail -20
```

**"Address already in use" error:**
```bash
# Find what's using the port
lsof -nP -iTCP:28787 -sTCP:LISTEN

# Kill it if necessary (be careful!)
kill <PID>
```

**Remote service not running:**
```bash
# Check if remote port is listening
ssh -p <PORT> <HOST> "lsof -nP -iTCP:8787 -sTCP:LISTEN"

# Start remote service if needed
ssh -p <PORT> <HOST> "cd ~/swe/vscode-shims && python launchers/launch_server_tmux.py"
```

---

## Troubleshooting

### "node: command not found"

**Cause:** Node.js not installed or not in PATH for non-interactive SSH.

**Fix:**
```bash
# Install Node.js
ssh <HOST> "/opt/homebrew/bin/brew install node"

# Always export PATH in SSH commands
ssh <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && node --version"
```

### "Failed to spawn CLI: [Errno 2] No such file or directory: 'node'"

**Cause:** Shim server can't find `node` executable because `/opt/homebrew/bin` is not in PATH when the Python process starts. This error appears in the webview UI when trying to start a Claude CLI session.

**Symptoms:**
- Webview loads successfully (http://localhost:28787 shows interface)
- Red error banner: "Failed to spawn CLI: [Errno 2] No such file or directory: 'node'"
- Shim HTTP logs show successful requests but no CLI spawn

**Fix:** Restart shim with PATH exported:
```bash
# WRONG - Python process inherits limited PATH
ssh <HOST> "cd ~/swe/vscode-shims && nohup python src/claude/server.py > /tmp/claude-shim.log 2>&1 &"

# CORRECT - Export PATH before starting shim
ssh <HOST> "pkill -f 'python.*claude.*server.py' && sleep 2 && \
  cd ~/swe/vscode-shims && \
  export PATH=\"/opt/homebrew/bin:\$PATH\" && \
  nohup ~/anaconda3/bin/python src/claude/server.py > /tmp/claude-shim.log 2>&1 &"
```

**Prevention:** Always include PATH export when starting shims remotely.

**Critical for:** cm2, cm3u, and any remote machine where `/opt/homebrew/bin` is not in default PATH.

### "python: command not found"

**Cause:** macOS has `python3` but no `python` alias.

**Fix:**
```bash
# Use python3 explicitly
ssh <HOST> "python3 --version"

# Fix scripts that use 'python'
sed -i '' 's/python /python3 /g' script.sh
```

### "Direct publishing is not allowed" (npm install)

**Cause:** Claude Code package.json has protection script.

**Fix:**
```bash
AUTHORIZED=1 npm install
```

### "CLI not found: /Users/sotola/swe/claude-code-2.1.12/cli.js"

**Cause:** Hardcoded path in server.py or missing env var.

**Fix:**
```bash
# Set in .env
echo "CLAUDE_CODE_CLI=~/symlinks/claude" >> ~/swe/vscode-shims/.env

# Or use --cli flag
python3 server.py --cli ~/symlinks/claude
```

### "No such file or directory: '/Users/sotola/agent-home'"

**Cause:** SHIM_DEFAULT_CWD points to non-existent directory.

**Fix:**
```bash
mkdir -p ~/agent-home
```

### "ModuleNotFoundError: No module named 'requests'"

**Cause:** Python dependencies not installed.

**Fix:**
```bash
pip3 install requests
```

### Shim server not accessible from LAN

**Cause:** Bound to localhost (127.0.0.1).

**Fix:** Set SHIM_HOST to your LAN IP in .env:
```bash
SHIM_HOST=192.168.1.9  # Replace with actual LAN IP
```

**Security note:** NEVER use `0.0.0.0` - it exposes to ALL networks.

### Auth errors after deployment

**Cause:** Credentials not copied or expired.

**Fix (Claude Code):**
```bash
# Automated upload to all remotes (recommended)
python ~/.claude/skills/claude-usage-meter/scripts/upload_credentials_to_remotes.py

# Manual upload via SCP
scp -P <PORT> ~/.claude/.credentials.json <HOST>:~/.claude/.credentials.json

# Manual upload via SSH alias
cat ~/.claude/.credentials.json | cm3u 'cat > ~/.claude/.credentials.json && chmod 600 ~/.claude/.credentials.json'
```

**Fix (Codex):**
```bash
# Copy auth
scp -P <PORT> ~/.codex/auth.json <HOST>:~/.codex/auth.json
```

---

## 🚨 CRITICAL: crypto.randomUUID Failure on HTTP/IP Access

<!-- [Added by Claude: 1446bf2e-3669-4050-a5ba-48be09357f30] -->
<!-- Rationale: This breaks remote webview completely - agents must know this immediately -->

**If remote webview messages fail silently, THIS IS LIKELY THE CAUSE.**

### Symptoms

- UI works, shows "Agent running"
- `POST /api/send` returns 200 OK
- Traffic logs show malformed messages (`"command": "sendMessage"`) instead of `io_message`
- Agent HQ shows brief flashing activity but no text appears

### Root Cause

`crypto.randomUUID()` is **unavailable** in insecure contexts (HTTP over IP like `http://192.168.1.9:8787`). Modern browsers disable it outside HTTPS/localhost.

When `randomUUID()` fails, the frontend falls back to a legacy payload format the server doesn't recognize → messages silently dropped.

### Fix: Inject Polyfill into vscode-shim.js

Add this at the **top** of `src/claude/public/vscode-shim.js` on the remote server:

```javascript
/* Polyfill for insecure contexts (HTTP over IP) */
if (!window.crypto) window.crypto = {};
if (!window.crypto.randomUUID) {
  window.crypto.randomUUID = function () {
    return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(
      /[xy]/g,
      function (c) {
        var r = (Math.random() * 16) | 0,
          v = c == "x" ? r : (r & 0x3) | 0x8;
        return v.toString(16);
      },
    );
  };
}
```

### Verification

1. Reload page with cache bust: `?v=new`
2. Open console, run: `typeof crypto.randomUUID` → should return `"function"`
3. Send a test message and verify it appears in logs

### ⚠️ WARNING: Do NOT use page.evaluate()

Injecting via devtools/page.evaluate only fixes the **running session**. On page reload, the fix is lost. **Always patch the file on disk.**

**📖 Full incident report**: See `references/incident_report_randomuuid.md` for the complete 2026-01-24 debugging story.

---

## ⚠️ CRITICAL: Never Run Electron on Remote

<!-- [Added by Claude: 1446bf2e-3669-4050-a5ba-48be09357f30] -->
<!-- Rationale: Electron apps spawn zombie processes that are hard to kill remotely -->

**On remote deployments, ONLY run the web version of Agent HQ.**

### Why?

- Electron apps spawn multiple background processes
- Over SSH, these become orphaned/zombie processes
- They accumulate and consume resources
- Require `pkill -9` or `kill -9` to terminate
- No benefit over web version when accessing remotely anyway

### Correct Approach

```bash
# WRONG - don't run Electron on remote
npm run dev        # Spawns Electron
npm run start      # Spawns Electron

# CORRECT - run web version only
npm run dev:web    # Web server only, access via browser
```

### If You Already Have Zombie Electron Processes

```bash
# Find them
ssh -p <PORT> <HOST> "ps aux | grep -i electron | grep -v grep"

# Kill them
ssh -p <PORT> <HOST> "pkill -9 -f 'Electron\|electron'"
```

---

## Directory Structure Reference

After complete deployment, the remote machine should have:

```
~/
├── symlinks/
│   ├── claude -> ~/swe/claude-code-X.Y.Z/cli.js
│   ├── claude-code-X.Y.Z -> ~/swe/claude-code-X.Y.Z
│   └── codex -> ~/swe/codex.X.Y.Z/codex-rs/target/release/codex
├── swe/
│   ├── claude-code-X.Y.Z/
│   │   ├── cli.js
│   │   ├── node_modules/
│   │   └── ...
│   ├── codex.X.Y.Z/
│   │   └── codex-rs/
│   │       └── target/release/codex
│   ├── vscode-shims/
│   │   ├── .env
│   │   ├── src/claude/server.py
│   │   └── launchers/launch_server.sh
│   └── telemetry_projects/ (optional)
├── agent-home/
│   ├── CLAUDE.md
│   └── AGENTS.md -> CLAUDE.md
├── AgenticProjects/
│   └── agent-box-v1/ (optional)
├── .claude/
│   └── .credentials.json
└── .codex/
    └── auth.json
```

---

## Environment Variables Reference

### vscode-shims .env

| Variable | Default | Purpose |
|----------|---------|---------|
| `SHIM_PORT` | `8787` | Claude shim HTTP port |
| `SHIM_HOST` | `127.0.0.1` | Network interface to bind (use LAN IP for network access) |
| `SHIM_DEFAULT_CWD` | `~/agent-home` | Working directory for spawned CLI |
| `CLAUDE_CODE_CLI` | (hardcoded fallback) | Path to Claude Code cli.js |
| `CODEX_SHIM_PORT` | `9288` | Codex shim HTTP port |
| `CODEX_SHIM_HOST` | `127.0.0.1` | Codex shim network interface |
| `CODEX_CLI` | (hardcoded fallback) | Path to Codex binary |

---

## Prerequisites Checklist

| Requirement | Check Command | Install Command |
|-------------|---------------|-----------------|
| Node.js | `node --version` | `/opt/homebrew/bin/brew install node` |
| npm | `npm --version` | (comes with Node.js) |
| Cargo/Rust | `cargo --version` | `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \| sh` |
| Python 3 | `python3 --version` | Usually pre-installed on macOS |
| Homebrew | `brew --version` | See brew.sh |

---

## 7. ~/.claude and ~/.codex Config Deployment

<!-- [Added by Claude: 1446bf2e-3669-4050-a5ba-48be09357f30] -->
<!-- Rationale: Skills and CLAUDE.md/AGENTS.md are essential for agent functionality -->

Deploy user-level config (CLAUDE.md, AGENTS.md) and skills to remote.

### ⚠️ IMPORTANT: Use Remote-Specific Config Files

Local CLAUDE.md and AGENTS.md contain local-only settings (say notifications, mirror-and-view script, VS Code troubleshooting, etc.) that don't apply to remotes.

**Use these pre-cleaned remote versions:**
- `~/.claude/CLAUDE.md.remote` → deploy as `~/.claude/CLAUDE.md`
- `~/.codex/AGENTS.md.remote` → deploy as `~/.codex/AGENTS.md`

**Sections removed in remote versions:**
- "say" command notifications (macOS audio)
- Mirror-and-view script
- Launcher Apps table
- VS Code Extension Troubleshooting
- Skill/Jutsu Archive (`~/KnowledgeBase`)
- Browser MCP note

**Sections preserved:**
- Agentic Log Files (observability is core to ecosystem)
- CLI aliases (consistency across machines)

### Quick Deploy (Config Files Only)

```bash
# Deploy CLAUDE.md and AGENTS.md to remote
scp -P <PORT> ~/.claude/CLAUDE.md.remote <HOST>:~/.claude/CLAUDE.md
scp -P <PORT> ~/.codex/AGENTS.md.remote <HOST>:~/.codex/AGENTS.md
```

### Handle Symlinks Properly (for Skills)

Before copying skills, check for symlinks in skills directories:

```bash
# Find symlinks in ~/.claude and ~/.codex
find ~/.claude -type l -ls 2>/dev/null
find ~/.codex -type l -ls 2>/dev/null
```

**Common symlinks to handle:**
- `~/.codex/skills/config-transparency-centralized` → `~/.claude/skills/...` (copy actual content)
- `~/.codex/skills/safe-port-kill` → `/tmp/...` (skip - not portable)

### Prepare and Transfer (Full Config + Skills)

```bash
# Prepare ~/.claude config (use .remote file, follow symlinks for skills)
mkdir -p /tmp/claude-config /tmp/codex-config
cp ~/.claude/CLAUDE.md.remote /tmp/claude-config/CLAUDE.md
rsync -aL ~/.claude/skills/ /tmp/claude-config/skills/

# Prepare ~/.codex config (use .remote file, handle symlinks manually for skills)
cp ~/.codex/AGENTS.md.remote /tmp/codex-config/AGENTS.md
cd ~/.codex/skills && for d in */; do
  name="${d%/}"
  if [ "$name" != "safe-port-kill" ]; then  # Skip broken symlinks
    cp -rL "$name" /tmp/codex-config/skills/
  fi
done
cp ~/.codex/skills/*.md /tmp/codex-config/skills/ 2>/dev/null

# Zip and transfer
cd /tmp && zip -rq claude-config.zip claude-config && zip -rq codex-config.zip codex-config
scp -P <PORT> /tmp/claude-config.zip /tmp/codex-config.zip <HOST>:/tmp/
```

### Deploy on Remote

```bash
ssh -p <PORT> <HOST> "cd /tmp && unzip -q claude-config.zip && unzip -q codex-config.zip && \
  mkdir -p ~/.claude/skills ~/.codex/skills && \
  cp /tmp/claude-config/CLAUDE.md ~/.claude/ && \
  rsync -a /tmp/claude-config/skills/ ~/.claude/skills/ && \
  cp /tmp/codex-config/AGENTS.md ~/.codex/ && \
  rsync -a /tmp/codex-config/skills/ ~/.codex/skills/"
```

### Initialize Git Repos (for tracking changes)

```bash
# ~/.claude git repo
ssh -p <PORT> <HOST> "cd ~/.claude && git init && \
  cat > .gitignore << 'EOF'
*
!.gitignore
!CLAUDE.md
!skills/
!skills/**
EOF
git add .gitignore CLAUDE.md skills/ && \
git commit -m 'Initial commit - CLAUDE.md and skills'"

# ~/.codex git repo
ssh -p <PORT> <HOST> "cd ~/.codex && git init && \
  cat > .gitignore << 'EOF'
*
!.gitignore
!AGENTS.md
!skills/
!skills/**
EOF
git add .gitignore AGENTS.md skills/ && \
git commit -m 'Initial commit - AGENTS.md and skills'"
```

---

## 8. Shell Aliases Deployment

<!-- [Added by Claude: 1446bf2e-3669-4050-a5ba-48be09357f30] -->
<!-- Rationale: Aliases like ccodex provide consistent CLI experience across machines -->

### Check Local Aliases

```bash
grep -E "^alias ccodex" ~/.zshrc
# Example output:
# alias ccodex='~/symlinks/codex --dangerously-bypass-approvals-and-sandbox ...'
# alias ccodex-low='~/symlinks/codex ... -c model_reasoning_effort=low ...'
# alias ccodex-instant='~/symlinks/codex ... -c model_reasoning_effort=none ...'
```

### Update Remote ~/.zshrc

```bash
# Download remote zshrc
scp -P <PORT> <HOST>:~/.zshrc /tmp/remote-zshrc

# Check for existing aliases
grep -E "^alias ccodex" /tmp/remote-zshrc

# Edit to add/update aliases (or use sed)
# Then upload back
scp -P <PORT> /tmp/remote-zshrc <HOST>:~/.zshrc
```

### Required Codex Aliases

```bash
alias ccodex='~/symlinks/codex --dangerously-bypass-approvals-and-sandbox --enable steer -c features.skills=true -c model_reasoning_effort=xhigh -c model=gpt-5.2'
alias ccodex-low='~/symlinks/codex --dangerously-bypass-approvals-and-sandbox -c features.skills=true -c model_reasoning_effort=low -c model=gpt-5.2'
alias ccodex-instant='~/symlinks/codex --dangerously-bypass-approvals-and-sandbox -c features.skills=true -c model_reasoning_effort=none -c model=gpt-5.2'
```

### Verify

```bash
ssh -p <PORT> <HOST> "source ~/.zshrc && alias | grep ccodex"
```

---

## 9. Git Integration for Remote Repos

<!-- [Added by Claude: 1446bf2e-3669-4050-a5ba-48be09357f30] -->
<!-- Rationale: .git enables git diff on remote to track drift from local committed state -->

### Why Include .git?

Without `.git`, you cannot run `git diff` on remote to see what's changed. The workflow:

1. **Local**: Commit all changes first
2. **Transfer**: Include `.git` in the zip (but exclude node_modules, target)
3. **Remote**: Overlay existing remote files onto unzipped copy
4. **Diff**: Now `git diff` shows remote drift vs local committed state

### Transfer with .git (Modified Workflow)

```bash
# On LOCAL - zip WITH .git but WITHOUT large dirs
cd ~/swe && zip -rq /tmp/project.zip project-name \
  -x "project-name/node_modules/*" \
  -x "project-name/target/*" \
  -x "project-name/.electron-dev/*" \
  -x "project-name/__pycache__/*"

# Transfer
scp -P <PORT> /tmp/project.zip <HOST>:/tmp/

# On REMOTE - unzip to /tmp first (don't overwrite existing yet)
ssh -p <PORT> <HOST> "mkdir -p /tmp/swe && cd /tmp/swe && unzip -q /tmp/project.zip"

# Overlay existing remote files onto /tmp copy (preserves .git)
ssh -p <PORT> <HOST> "rsync -a --exclude='.git' ~/swe/project-name/ /tmp/swe/project-name/"

# Check git diff
ssh -p <PORT> <HOST> "cd /tmp/swe/project-name && git diff --stat"

# If satisfied, copy back to final location
ssh -p <PORT> <HOST> "rsync -a /tmp/swe/project-name/ ~/swe/project-name/"
```

---

## 10. Deployment Logs

<!-- [Added by Claude: 1446bf2e-3669-4050-a5ba-48be09357f30] -->
<!-- Rationale: Document what was deployed for future reference and audit trail -->

Create `deployment_logs/` directory in each deployed project:

```bash
ssh -p <PORT> <HOST> "mkdir -p ~/swe/project-name/deployment_logs"
```

### Log File Format

Create `deployment_logs/YYYY_MM_DD.md`:

```markdown
# Deployment Log - YYYY-MM-DD

## Summary
Successfully deployed project-name to remote.

## Agents Involved
| Agent ID | Type | Role |
|----------|------|------|
| abc123... | Claude | Initial deployment |
| def456... | Claude | Git integration |

## Deployment Steps
1. Local commit: `abc1234` - "Commit message"
2. Transfer: Zipped (X MB) and transferred
3. Git integration: Unzipped with .git, overlay, diff

## Key Changes Deployed
- Feature A
- Fix B
- Config C

## Remote Config Differences
- .env: Network bindings differ
- etc.

## Verification
\`\`\`bash
git log --oneline -1
\`\`\`
```

### Commit Deployment Logs

```bash
ssh -p <PORT> <HOST> "cd ~/swe/project-name && \
  git add deployment_logs/ && \
  git commit -m 'Deployment successful - see YYYY_MM_DD.md for details'"
```

---

## Updated Directory Structure Reference

After **complete** deployment, the remote machine should have:

```
~/
├── symlinks/
│   ├── claude -> ~/swe/claude-code-X.Y.Z/cli.js
│   └── codex -> ~/swe/codex.X.Y.Z/codex-rs/target/release/codex
├── swe/
│   ├── claude-code-X.Y.Z/
│   │   ├── .git/                    # NEW: git history
│   │   ├── deployment_logs/         # NEW: deployment docs
│   │   ├── cli.js
│   │   └── node_modules/
│   ├── codex.X.Y.Z/
│   │   ├── .git/                    # NEW: git history
│   │   ├── deployment_logs/         # NEW: deployment docs
│   │   └── codex-rs/target/release/codex
│   ├── vscode-shims/
│   │   ├── .git/                    # NEW: git history
│   │   ├── deployment_logs/         # NEW: deployment docs
│   │   └── ...
│   └── telemetry_projects/ (optional)
├── AgenticProjects/
│   └── agent-box-v1/
│       ├── .git/                    # NEW: git history
│       ├── deployment_logs/         # NEW: deployment docs
│       └── ...
├── .claude/
│   ├── .git/                        # NEW: git history
│   ├── CLAUDE.md
│   ├── skills/                      # NEW: all skills
│   └── .credentials.json
├── .codex/
│   ├── .git/                        # NEW: git history
│   ├── AGENTS.md
│   ├── skills/                      # NEW: all skills
│   └── auth.json
└── .zshrc                           # UPDATED: ccodex aliases
```

---

## Complete Deployment Checklist

<!-- [Added by Claude: 1446bf2e-3669-4050-a5ba-48be09357f30] -->

Use this checklist to verify a complete deployment:

- [ ] **Prerequisites**: Node.js, Cargo, Python3, Homebrew installed
- [ ] **Claude Code**: Deployed with .git, symlink created
- [ ] **Codex CLI**: Deployed with .git, built, symlink created
- [ ] **vscode-shims**: Deployed with .git, .env configured
- [ ] **agent-box-v1**: Deployed with .git (optional)
- [ ] **~/.claude**: CLAUDE.md + skills deployed, git initialized
- [ ] **~/.codex**: AGENTS.md + skills deployed, git initialized
- [ ] **Auth**: .credentials.json and auth.json copied
- [ ] **Shell aliases**: ccodex, ccodex-low, ccodex-instant in ~/.zshrc
- [ ] **Deployment logs**: Created in each project
- [ ] **Smoke tests**: CLI versions verified, aliases working

---

## 🚨 Incident: Agent HQ UI Crash (2026-01-25)

<!-- [Added by Claude: 3ae26ab6-64d2-4932-8ed8-4ca6f4313658 2026-01-25] -->

### Symptom

- `http://localhost:28037` shows "This site can't be reached" / "ERR_CONNECTION_RESET"
- Port 8037 not listening on remote
- AMS panel shows "SSE connection lost"

### Root Cause

Agent HQ UI (vite) requires environment variables from `.env`:
- `AMS_TMUX_PORT` (or `AGENT_MGMT_PORT`)
- `AGENT_HQ_UI_PORT` (or `VITE_PORT`)

**The original deployment started Agent HQ without sourcing `.env`**, so the process crashed with:
```
Error: Missing/invalid AMS port: set AMS_TMUX_PORT (preferred) or AGENT_MGMT_PORT
Error: Missing/invalid port: set AGENT_HQ_UI_PORT (preferred) or VITE_PORT
```

### Fix

**Must source `.env` before starting vite:**

```bash
ssh -p <PORT> <HOST> "export PATH=\"/opt/homebrew/bin:\$PATH\" && \
  tmux new-session -d -s agent-hq-ui-8037 \
  -c ~/AgenticProjects/agent-box-v1/apps/agent-hq-ui \
  'bash -lc \"cd ~/AgenticProjects/agent-box-v1 && set -a && source .env && set +a && \
  cd apps/agent-hq-ui && npm run dev:web -- --port 8037 --host 127.0.0.1; exec bash\"'"
```

**Key pattern:** `set -a && source .env && set +a` exports all variables from `.env` to the environment.

### Prevention

Always start Agent HQ services by:
1. Sourcing the repo root `.env` first
2. Running in a persistent tmux session
3. Using `bash -lc` to get a login shell with proper PATH

See skill: **remote-agent-hq-startup** for complete troubleshooting guide.

---

## OAuth Credential Management

**When to refresh and upload credentials:**

1. **Authentication failures** on remote machines
2. **Deploying to a new remote** machine for the first time
3. **After `/login` on local** machine (to sync to remotes)
4. **Periodic refresh** (every 30-90 days to stay ahead of token expiration)

**Quick workflow:**

```bash
# 1. Generate fresh OAuth (interactive TUI)
mkdir -p /tmp/agent-$(uuidgen | tail -c 6)
CLAUDE_CONFIG_DIR=/tmp/agent-XXXXX CLAUDE_CODE_ENABLE_OAUTH_TOKEN_DUMP=1 ~/swe/claude-code-2.0.28/cli.js
# Type /login in TUI, complete OAuth flow

# 2. Convert to credentials format
python ~/.claude/skills/claude-usage-meter/scripts/convert_oauth_to_credentials.py \
  --config-dir /tmp/agent-XXXXX --output ~/.claude/.credentials.json

# 3. Upload to all remotes
python ~/.claude/skills/claude-usage-meter/scripts/upload_credentials_to_remotes.py
```

**Script location:** `~/.claude/skills/claude-usage-meter/scripts/upload_credentials_to_remotes.py`

**How it works:**
- Parses SSH aliases from `~/.zshrc` (cm3u, cm2, etc.)
- Pipes credentials securely via SSH stdin
- Sets correct permissions (600) on remote
- Tests each remote with `~/symlinks/claude -p "test"`
- Reports success/failure for each remote

---

## Related Skills

- **claude-usage-meter** - OAuth token refresh and usage API (includes upload_credentials_to_remotes.py script)
- **codex-remote-auth** - Codex authentication deployment
- **config-transparency-centralized** - .env configuration philosophy
- **search-agent-conversation** - Find agents who worked on deployment
- **remote-agent-hq-startup** - Debug AMS/Agent HQ startup issues