Hotplex Diagnostics

Docker & HotPlex Diagnostics

Comprehensive diagnostic system for Docker containers with enhanced capabilities for HotPlex services.

Diagnostic Layers

Quick Start

Full System Scan (All Containers)

# Layer 1: Quick health of ALL containers
docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | head -20

# Layer 2: Resource usage of ALL containers
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"

HotPlex-Only Quick Check

cd ~/hotplex/docker/matrix && docker compose ps

Layer 1: Universal Container Health

List All Running Containers

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"

List All Containers (Including Stopped)

docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

Health Status Check (All Containers with Healthchecks)

docker ps --format "{{.Names}}" | while read c; do
  health=$(docker inspect "$c" --format='{{.State.Health.Status}}' 2>/dev/null || echo "n/a")
  printf "%-30s %s\n" "$c" "$health"
done

Container State Summary

echo "=== Container State Summary ===" && \
docker ps -a --format "{{.State}}" | sort | uniq -c | sort -rn

Recent Container Events (Restarts, Deaths)

docker events --since 1h --filter 'type=container' --format '{{.Action}} {{.Actor.Attributes.name}}' 2>/dev/null | tail -20

Layer 2: Resource & Log Analysis

Resource Usage (All Containers)

# CPU & Memory
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Top CPU consumers
docker stats --no-stream --format "{{.Name}}\t{{.CPUPerc}}" | sort -t$'\t' -k2 -rn | head -5

# Top Memory consumers
docker stats --no-stream --format "{{.Name}}\t{{.MemPerc}}" | sort -t$'\t' -k2 -rn | head -5

Network I/O Analysis

docker stats --no-stream --format "table {{.Name}}\t{{.NetIO}}\t{{.BlockIO}}"

Error Scan (All Containers)

echo "=== Recent Errors (All Containers) ===" && \
for c in $(docker ps --format "{{.Names}}"); do
  errors=$(docker logs "$c" --since 1h 2>&1 | grep -iE "error|fatal|panic|exception" | head -3)
  if [ -n "$errors" ]; then
    echo "--- $c ---"
    echo "$errors"
  fi
done

Container Restart History

for c in $(docker ps -a --format "{{.Names}}"); do
  restarts=$(docker inspect "$c" --format='{{.RestartCount}}' 2>/dev/null || echo "0")
  if [ "$restarts" -gt 0 ]; then
    echo "$c: $restarts restarts"
  fi
done

Layer 3: HotPlex Enhanced Diagnostics

Working Directory: ~/hotplex/docker/matrix

Container Discovery

IMPORTANT: Do not hardcode container details. Always discover containers dynamically:

# Discover HotPlex containers
cd ~/hotplex/docker/matrix && docker compose ps

# Get container names and ports
docker ps --format "table {{.Names}}\t{{.Ports}}" | grep hotplex

Port Mapping Convention

HotPlex follows a predictable port numbering pattern:

• Main Port (WebSocket/HTTP): 18080 + (BOT_INDEX - 1)
• Admin Port (Session Management): 19080 + (BOT_INDEX - 1)

Examples:

• Bot 01: Main=18080, Admin=19080
• Bot 02: Main=18081, Admin=19081
• Bot 03: Main=18082, Admin=19082

However, always verify actual ports using docker compose ps rather than assuming.

HTTP Health Check

# Discover HotPlex containers and check health
cd ~/hotplex/docker/matrix && \
for container in $(docker compose ps --services); do
  port=$(docker port "${container}_1" 8080 2>/dev/null | grep -oP ':\K\d+' || echo "N/A")
  if [ "$port" != "N/A" ]; then
    status=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:$port/health 2>/dev/null || echo "000")
    echo "$container (port $port): HTTP $status"
  fi
done

# External services (example - only if running)
if docker ps --format '{{.Names}}' | grep -q openclaw-gateway; then
  echo "OpenClaw Gateway:"
  curl -s http://localhost:18789/health 2>/dev/null || echo "  FAILED or not running"
fi

Socket Mode Status

cd ~/hotplex/docker/matrix && \
docker compose logs --tail=100 2>&1 | grep -E "Socket Mode|invalid_auth|Connected|Disconnected"

Session Diagnostics

# Discover containers and check active sessions
cd ~/hotplex/docker/matrix && \
for container in $(docker compose ps --services); do
  instance="${container}_1"
  active=$(docker exec "$instance" ps aux 2>/dev/null | grep -cE "claude|opencode" || echo "0")
  echo "$container: $active active sessions"
done

# Session count for specific container (example)
# docker logs hotplex-01 --since 1h 2>&1 | grep -c "session_id"

HotPlex-Specific Errors

cd ~/hotplex/docker/matrix && \
for container in $(docker compose ps --services); do
  instance="${container}_1"
  echo "=== $container ==="
  docker logs "$instance" --since 1h 2>&1 | grep -iE "error|waf|blocked|timeout" | head -5
done

Disk Usage (HotPlex)

# Check disk usage for all HotPlex containers
cd ~/hotplex/docker/matrix && \
for container in $(docker compose ps --services); do
  instance="${container}_1"
  echo "=== $container ==="
  docker exec "$instance" du -sh /home/hotplex/.hotplex 2>/dev/null || echo "  N/A"
  docker exec "$instance" du -sh /home/hotplex/.claude 2>/dev/null || echo "  N/A"
done

OpenClaw Gateway Diagnostics

Note: openclaw-gateway runs independently (not in docker-compose.yml).

# Health check
curl -s http://localhost:18789/health | jq

# Container status
docker ps --filter "name=openclaw-gateway" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Recent logs
docker logs openclaw-gateway --tail=100 2>&1

# Error scan
docker logs openclaw-gateway --since 1h 2>&1 | grep -iE "error|fatal|panic|exception"

# Resource usage
docker stats --no-stream openclaw-gateway --format "CPU: {{.CPUPerc}}, Memory: {{.MemUsage}}"

# If container not found, check if it's supposed to be running
echo "Note: openclaw-gateway is not managed by docker-compose."
echo "It may be started by a separate compose file or manually."

Admin CLI Commands

HotPlex provides admin CLI commands for quick diagnostics:

Quick Status Check

# Check daemon status (from any bot container)
cd ~/hotplex/docker/matrix && \
for container in $(docker compose ps --services | head -1); do
  docker exec "${container}_1" hotplexd status
done

Diagnostic Checks

# Run doctor diagnostic (from any bot container)
cd ~/hotplex/docker/matrix && \
for container in $(docker compose ps --services | head -1); do
  echo "=== Running doctor in $container ==="
  docker exec "${container}_1" hotplexd doctor
done

Doctor checks:

• CLI binary availability (claude-code)
• Configuration file validity
• Required environment variables
• Port availability (8080, 9080)
• Database connectivity (if configured)

Config Validation

# Validate configuration file (from any bot container)
cd ~/hotplex/docker/matrix && \
container=$(docker compose ps --services | head -1) && \
docker exec "${container}_1" hotplexd config validate /home/hotplex/.hotplex/config.yaml

Diagnostic Workflows

Workflow 1: Full System Health Report

Use when user asks for general system health or "check everything":

echo "=== Docker System Health Report ===" && \
echo "" && \
echo ">>> Container States" && \
docker ps -a --format "{{.State}}" | sort | uniq -c && \
echo "" && \
echo ">>> Resource Usage (Top 10)" && \
docker stats --no-stream --format "{{.Name}}\t{{.CPUPerc}}\t{{.MemPerc}}" | head -11 && \
echo "" && \
echo ">>> Recent Errors" && \
docker ps --format "{{.Names}}" | head -5 | while read c; do
  e=$(docker logs "$c" --since 30m 2>&1 | grep -ciE "error|fatal")
  [ "$e" -gt 0 ] && echo "$c: $e errors"
done

Workflow 2: Container-Specific Deep Dive

Use when user asks about a specific container:

CONTAINER="<container-name>"

echo "=== Deep Dive: $CONTAINER ===" && \
echo "" && \
echo ">>> Status" && \
docker inspect "$CONTAINER" --format='Status: {{.State.Status}}, Health: {{.State.Health.Status}}, Restarts: {{.RestartCount}}' 2>/dev/null && \
echo "" && \
echo ">>> Resources" && \
docker stats --no-stream "$CONTAINER" --format "CPU: {{.CPUPerc}}, Memory: {{.MemUsage}}" 2>/dev/null && \
echo "" && \
echo ">>> Recent Logs (Last 50 lines)" && \
docker logs "$CONTAINER" --tail=50 2>&1

Workflow 3: HotPlex Cluster Health

Use when user asks about HotPlex specifically:

echo "=== HotPlex Cluster Health ===" && \
cd ~/hotplex/docker/matrix && \
echo "" && \
echo ">>> Container Status" && \
docker compose ps && \
echo "" && \
echo ">>> HTTP Health" && \
echo "Main servers:" && \
for container in $(docker compose ps --services); do
  port=$(docker port "${container}_1" 8080 2>/dev/null | grep -oP ':\K\d+')
  if [ -n "$port" ]; then
    status=$(curl -s http://localhost:$port/health 2>/dev/null || echo "FAILED")
    echo "$container (port $port): $status"
  fi
done && \
echo "Admin servers:" && \
for container in $(docker compose ps --services); do
  admin_port=$(docker port "${container}_1" 9080 2>/dev/null | grep -oP ':\K\d+')
  if [ -n "$admin_port" ]; then
    status=$(curl -s http://localhost:$admin_port/admin/v1/stats 2>/dev/null || echo "FAILED")
    echo "$container (port $admin_port): $status"
  fi
done && \
echo "" && \
echo ">>> Resource Usage" && \
docker stats --no-stream $(docker compose ps -q) --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}" 2>/dev/null

Common Container Health Indicators

Reference Documentation

For deeper diagnostics, consult these reference files:

• references/container-health.md - Universal container health indicators and thresholds
• references/hotplex-deep-dive.md - HotPlex-specific diagnostic procedures
• references/api-endpoints.md - HotPlex WebSocket and HTTP API reference

Diagnostic Script

For automated comprehensive diagnostics:

# Run full diagnostic scan
~/.agent/skills/hotplex-diagnostics/scripts/diagnose-all.sh all

# Specific layers
~/.agent/skills/hotplex-diagnostics/scripts/diagnose-all.sh 1     # Health only
~/.agent/skills/hotplex-diagnostics/scripts/diagnose-all.sh 2     # Resources only
~/.agent/skills/hotplex-diagnostics/scripts/diagnose-all.sh 3     # Logs only

# Deep dive on specific container
~/.agent/skills/hotplex-diagnostics/scripts/diagnose-all.sh <container-name>

Related Skills

• docker-container-ops - Container lifecycle management
• hotplex-data-mgmt - Data and session management

Source: hrygo/hotplex-legacy — distributed by TomeVault.