Production Debugging

Production Server Debugging

Debug and manage the Orient production environment on Oracle Cloud.

Prerequisites

SSH access requires these environment variables (stored in .env):

• OCI_HOST - Oracle Cloud server IP (default: 152.70.172.33)
• OCI_USER - SSH username (default: opc)
• SSH key configured in ~/.ssh/id_rsa

Quick setup (if not in .env):

export OCI_HOST=152.70.172.33
export OCI_USER=opc

SSH Connection

Connect to Production

ssh -o StrictHostKeyChecking=no $OCI_USER@$OCI_HOST

Run Remote Command

ssh -o StrictHostKeyChecking=no $OCI_USER@$OCI_HOST "command"

Example: Check if containers are running

ssh $OCI_USER@$OCI_HOST "docker ps --format 'table {{.Names}}\t{{.Status}}'"

Server Directory Structure

~/orient/
├── docker/           # Compose files and configs
│   ├── docker-compose.yml
│   ├── docker-compose.v2.yml
│   ├── docker-compose.prod.yml
│   ├── docker-compose.r2.yml
│   ├── nginx-ssl.conf
│   └── deploy-server.sh
├── data/             # Persistent data (volumes)
├── logs/             # Application logs
├── backups/          # Deployment backups
└── .env              # Environment variables

Working directory for Docker commands:

cd ~/orient/docker

Container Overview

Container Logs

View Recent Logs

# Core containers
docker logs orienter-dashboard --tail 100
docker logs orienter-bot-whatsapp --tail 100
docker logs orienter-opencode --tail 100
docker logs orienter-nginx --tail 50

Follow Logs in Real-Time

docker logs -f orienter-dashboard

Logs Since Specific Time

docker logs orienter-dashboard --since 1h
docker logs orienter-dashboard --since "2024-01-07T10:00:00"

Parse JSON Logs

docker logs orienter-dashboard --tail 50 2>&1 | jq -R 'fromjson? // .'

Search for Errors

docker logs orienter-dashboard 2>&1 | grep -i error | tail -20

Quick Debugging Reference

Common Production Issues

1. Dashboard Container Crash Loop

Symptoms: Dashboard shows "Restarting" status, nginx returns 502

Check logs:

ssh $OCI_USER@$OCI_HOST "docker logs orienter-dashboard --tail 100 2>&1"

Common causes:

Express 5 / path-to-regexp Error

Error: TypeError: Missing parameter name at index 1: *

This happens when using bare * wildcards in Express 5 routes:

// BROKEN
app.get('*', (req, res) => { ... });

// FIXED
app.get('/{*splat}', (req, res) => { ... });

Fix: Update the SPA catch-all route in packages/dashboard/src/server/index.ts

Database Connection Error

Error: ECONNREFUSED or connection refused

# Check postgres is running
docker ps | grep postgres

# Check DATABASE_URL
docker exec orienter-dashboard env | grep DATABASE_URL

2. Dashboard Assets Not Loading (text/html error)

Symptoms: Browser console shows:

Failed to load module script: Expected a JavaScript-or-Wasm module script
but the server responded with a MIME type of "text/html"

Cause: Nginx is incorrectly proxying asset requests

Debug:

# Check what content-type is returned for JS files
curl -sI "https://app.orient.bot/dashboard/assets/index-*.js" | grep content-type

# Expected: content-type: text/javascript; charset=utf-8
# If: content-type: text/html -> nginx routing issue

3. 502 Bad Gateway

Check which service is down:

docker ps --format "table {{.Names}}\t{{.Status}}"

Check nginx upstream:

docker logs orienter-nginx --tail 20 | grep -i "upstream\|502"

Common causes:

• Dashboard not started (crash loop)
• Wrong port in nginx config
• Container name mismatch

4. WhatsApp Pairing Issues

Symptoms:

• Dashboard shows "Connection Closed" when requesting pairing code
• QR code stops refreshing
• Logs show QR refs attempts ended error

Quick fix:

ssh $OCI_USER@$OCI_HOST "docker restart orienter-bot-whatsapp"

Full reset:

ssh $OCI_USER@$OCI_HOST "rm -rf /home/opc/orient/data/whatsapp-auth/* && docker restart orienter-bot-whatsapp"

Diagnosis:

ssh $OCI_USER@$OCI_HOST "docker logs orienter-bot-whatsapp --tail 100 2>&1 | grep -i -E '(pair|code|connection|closed|error)'"

5. Missing Environment Variables (Crash Loop)

Symptoms: Container in crash loop with "Restarting" status, logs show missing environment variable errors

Example error:

Error: DASHBOARD_JWT_SECRET environment variable is required

Common missing variables:

• DASHBOARD_JWT_SECRET (production) or DASHBOARD_JWT_SECRET_STAGING (staging)
• DATABASE_URL
• OAuth credentials (GOOGLE_OAUTH_CLIENT_ID, etc.)
• API keys (Anthropic, OpenAI, etc.)

Quick checklist for staging environment:

# Check if variable exists in .env
ssh $OCI_USER@$OCI_HOST "grep DASHBOARD_JWT_SECRET_STAGING /home/opc/orient/.env"

# Check what env vars the container actually has
ssh $OCI_USER@$OCI_HOST "docker exec orienter-dashboard-staging env | grep DASHBOARD"

Critical knowledge about environment variables:

1. Staging uses _STAGING suffix: Staging containers expect environment variables with _STAGING suffix

   • Production: DASHBOARD_JWT_SECRET
   • Staging: DASHBOARD_JWT_SECRET_STAGING

2. docker restart does NOT reload .env: Simply restarting a container won't pick up new environment variables

   # WRONG - Won't reload env vars
   docker restart orienter-dashboard-staging
   
   # CORRECT - Recreates container with new env vars
   cd ~/orient/docker
   docker compose --env-file ../.env -f docker-compose.v2.yml -f docker-compose.staging.yml up -d dashboard
   

3. docker-compose needs --env-file flag: When .env is in parent directory, must specify explicitly

Fix missing environment variables:

# 1. Add missing variable to .env file
ssh $OCI_USER@$OCI_HOST "echo 'DASHBOARD_JWT_SECRET_STAGING=\"your-secret-here\"' >> /home/opc/orient/.env"

# 2. Recreate container with updated env (from docker directory)
ssh $OCI_USER@$OCI_HOST "cd /home/opc/orient/docker && docker compose --env-file ../.env -f docker-compose.v2.yml -f docker-compose.staging.yml up -d dashboard"

# 3. Verify container is healthy
ssh $OCI_USER@$OCI_HOST "docker ps | grep dashboard"

# 4. Check logs for successful startup
ssh $OCI_USER@$OCI_HOST "docker logs orienter-dashboard-staging --tail 20"

7. SSH Command Not Found (Monitoring Failure)

Symptoms: Monitoring page shows "Failed to load server metrics" with error logs showing:

/bin/sh: ssh: not found

Cause: The dashboard container doesn't have openssh-client installed. This is required for the monitoring service which SSHes into the production server to collect metrics.

Quick check:

# Verify SSH is available in container
docker exec orienter-dashboard which ssh
# Expected: /usr/bin/ssh
# If empty or "not found": SSH client missing

Fix for Dockerfile: The packages/dashboard/Dockerfile must include openssh-client in the alpine packages:

# In the runner stage
RUN apk add --no-cache curl bash openssh-client

Workaround for staging (build patched image locally):

# 1. Build a patched image with SSH support
cd ~/orient
docker build -t dashboard-staging-ssh -f packages/dashboard/Dockerfile .

# 2. Update staging compose to use local image
sed -i 's|image: ghcr.io/orient-code/orient/dashboard:staging|image: dashboard-staging-ssh|' docker/docker-compose.staging.yml

# 3. Recreate container
cd ~/orient/docker
docker compose --env-file ../.env -f docker-compose.v2.yml -f docker-compose.staging.yml up -d dashboard

8. SSH Key Mounting for Containers

Problem: Container needs SSH access to production server for monitoring, but SSH keys need proper permissions.

Docker Compose configuration:

# In docker-compose.v2.yml
dashboard:
  environment:
    - SSH_KEY_PATH=/app/.ssh/id_rsa
  volumes:
    # Mount SSH key from docker/.ssh directory (NOT ~/.ssh which has permission issues)
    - ./.ssh:/app/.ssh:ro

Setup steps on server:

# 1. Create docker/.ssh directory
mkdir -p ~/orient/docker/.ssh

# 2. Generate or copy SSH key (must be readable by container's non-root user)
ssh-keygen -t rsa -f ~/orient/docker/.ssh/id_rsa -N ""

# 3. Set readable permissions (container runs as uid 1001)
chmod 644 ~/orient/docker/.ssh/id_rsa
chmod 644 ~/orient/docker/.ssh/id_rsa.pub

# 4. Add public key to authorized_keys for localhost access
cat ~/orient/docker/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

# 5. Recreate container to pick up the mount
cd ~/orient/docker
docker compose --env-file ../.env -f docker-compose.v2.yml up -d dashboard

Why ./.ssh instead of ~/.ssh?

• Tilde (~) doesn't expand in docker-compose volume paths
• User's ~/.ssh typically has strict permissions (600) that block container access
• Container runs as non-root user (nodejs, uid 1001), needs readable permissions

Required environment variables (in .env):

OCI_HOST=152.70.172.33   # or SSH_HOST
OCI_USER=opc             # or SSH_USER
# SSH_KEY_PATH is set in compose file: /app/.ssh/id_rsa

Verify SSH works from container:

docker exec orienter-dashboard ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i /app/.ssh/id_rsa opc@152.70.172.33 'echo SSH works'

9. Container Environment Variable Reload

Critical: docker restart does NOT reload environment variables from .env. You must recreate the container.

Wrong approach (env vars won't update):

# This won't pick up new .env values!
docker restart orienter-dashboard

Correct approach (recreates container with fresh env):

cd ~/orient/docker
docker compose --env-file ../.env -f docker-compose.v2.yml -f docker-compose.prod.yml up -d dashboard

For staging containers:

cd ~/orient/docker
docker compose --env-file ../.env -f docker-compose.v2.yml -f docker-compose.staging.yml up -d dashboard

Verify environment variables loaded:

# Check specific variable
docker exec orienter-dashboard env | grep SSH_KEY_PATH
docker exec orienter-dashboard env | grep OCI_HOST

# For staging
docker exec orienter-dashboard-staging env | grep SSH_KEY_PATH

Common mistake with staging: Forgetting the _STAGING suffix for environment variables:

• Production uses: DASHBOARD_JWT_SECRET
• Staging uses: DASHBOARD_JWT_SECRET_STAGING

6. Deployment Failure

Check recent GitHub Actions:

gh run list --limit 5
gh run view <run-id>

Check which images were built:

gh run view <run-id> | grep -E "Build.*Image"

If builds show 0s with - prefix, they were skipped (no changes detected).

Force rebuild if needed:

gh workflow run deploy.yml -f force_build_all=true

Container Management

List Running Containers

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Restart a Container

docker restart orienter-dashboard
docker restart orienter-bot-whatsapp
docker restart orienter-opencode
docker restart orienter-nginx

Restart All Services

cd ~/orient/docker
COMPOSE_FILES="-f docker-compose.v2.yml -f docker-compose.prod.yml -f docker-compose.r2.yml"
docker compose ${COMPOSE_FILES} restart

Execute Command in Container

docker exec -it orienter-dashboard sh
docker exec -it orienter-opencode bash
docker exec orienter-postgres psql -U aibot -d whatsapp_bot

Check Container Files

# Check dashboard assets exist
docker exec orienter-dashboard ls -la /app/packages/dashboard/public/assets/

# Check nginx config
docker exec orienter-nginx cat /etc/nginx/conf.d/default.conf

View Container Resource Usage

docker stats --no-stream

Inspect Container

docker inspect orienter-dashboard | jq '.[0].State'

Health Checks

Verify All Services

# Nginx
curl -sf https://app.orient.bot/health && echo "Nginx: OK" || echo "Nginx: FAIL"

# Dashboard
curl -sf https://app.orient.bot/dashboard/api/health && echo "Dashboard: OK" || echo "Dashboard: FAIL"

# OpenCode
curl -sf https://code.orient.bot/global/health && echo "OpenCode: OK" || echo "OpenCode: FAIL"

Check Database

docker exec orienter-postgres pg_isready -U aibot -d whatsapp_bot

Deployment Management

Check Deployment Script Options

~/orient/docker/deploy-server.sh --help

Rollback to Previous Deployment

cd ~/orient/docker
./deploy-server.sh rollback

Manual Full Restart

cd ~/orient/docker
COMPOSE_FILES="-f docker-compose.v2.yml -f docker-compose.prod.yml -f docker-compose.r2.yml"
docker compose ${COMPOSE_FILES} down
docker compose ${COMPOSE_FILES} up -d

Check Disk Space

df -h /home
docker system df

Clean Up Docker Resources

docker system prune -f
docker image prune -a -f --filter "until=168h"  # Remove images older than 7 days

Error Pattern Reference

Session Data Locations

/home/opc/orient/data/
├── whatsapp-auth/     # WhatsApp session files
│   ├── creds.json     # Credentials
│   └── .pairing-mode  # Pairing mode marker
├── media/             # Uploaded media files
├── oauth-tokens/      # OAuth tokens
└── postgres/          # Database files (volume)

Log Locations

• Container logs: docker logs <container>
• Nginx access: docker exec orienter-nginx cat /var/log/nginx/access.log
• Application logs: ~/orient/logs/ (if configured)

Server Details

• Host: 152.70.172.33
• User: opc
• SSH Key: ~/.ssh/id_rsa
• Deploy Directory: ~/orient
• Docker Directory: ~/orient/docker
• Domains:
  • app.orient.bot - Dashboard
  • code.orient.bot - OpenCode
  • staging.orient.bot - Staging Dashboard
  • code-staging.orient.bot - Staging OpenCode