Federation Health

Federation Health

Three domains unified: Federation Orchestrator + VPS Telemetry Auto-Healer + Docker Entropy. Containers are thermodynamic systems: CPU is energy, memory is state, disk is storage, network is exchange. DITEMPA BUKAN DIBERI

Overview

Unified federation health covering three domains:

1. Federation Orchestrator — Monitor and manage the arifOS federation: docker compose ps, restart dead services, MCP health probes, container drift detection, restart count monitoring.
2. VPS Telemetry Auto-Healer — Real-time Caddy HTTP error spike monitor, Docker container log stream analyzer, and automated container health recovery watchdog.
3. Docker Entropy — Thermodynamic reasoning lens for container fleet health, resource pressure, and safe/dangerous intervention boundaries.

arifOS-ACT Embedding

Before using this skill on any mutating, irreversible, or high-blast-radius task:

1. ART — Attune (what is the real task?), Recognize (what class of power?), Test (fit · authority · evidence · blast · reversible).
2. Kernel — Route to arifOS for F1–F13 judgment if action class is Maker/Messenger/Mutator/Destroyer/Sovereign.
3. ACT — Apply narrow, Constrain scope, Trace witness, STOP before corruption.
4. Receipt — Leave evidence of what changed, why, and under whose authority.

When to Use

• A Docker container or compose stack appears unhealthy, restarting, or resource-starved.
• Disk pressure, memory pressure, or CPU saturation is suspected on the VPS.
• You need to decide whether to restart, prune, inspect logs, or escalate.
• An MCP server returns connection refused, timeout, or 5xx.
• Caddy HTTP 502/504 error spikes detected.
• Federation status report needed.
• Container restart count tracking (flag containers with >3 restarts).
• After cron changes to verify telemetry pipeline.

When NOT to Use

• Do not use for non-Docker services (systemd-native, bare-metal processes) — use systemd skills.
• Do not use for Kubernetes or Swarm orchestration (not deployed).
• Do not use for Docker image authoring, Dockerfile creation, or CI/CD pipeline configuration.
• Do not use as authority to run destructive commands without 888 HOLD.

Section 1: Federation Orchestrator

Commands

# Full federation status
docker compose ps
docker ps -a --filter "status=restarting"

# Check organ endpoints (verified 2026-08-14)
for url in 8088 7071 7072 7073 7074 8081 18082 18083; do
  curl -s -o /dev/null -w "%{http_code} :$url\n" --max-time 5 http://localhost:$url/health
done

# Restart dead container
docker compose restart <service>
docker restart <container>

# Restart count audit
docker inspect --format='{{.Name}} {{.RestartCount}}' $(docker ps -aq)

# Federation stack locations
cd /root/arifOS && docker compose ps
cd /root/AAA && docker compose ps

Health Probe Script

#!/bin/bash
for port in 8088 7071 7072 7073 7074 8081 18082 18083; do
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:$port/health)
  if [ "$code" = "200" ]; then
    echo "✅ MCP:$port OK"
  else
    echo "❌ MCP:$port FAILED (HTTP $code)"
  fi
done

Restart Count Thresholds

Observed Port Map (verified 2026-08-14 via ss + systemctl + curl)

Federation Nodes (LEGACY docker-compose era — verify at runtime before trusting)

Section 2: VPS Telemetry Auto-Healer

Diagnostic & Triage Workflow

1. Check Container Health & Memory Throttling

docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}"

2. Scan Caddy Log Stream for 502 / 504 Spikes

tail -n 200 /var/log/caddy/access.log | grep -E "status\":(502|504|500)"

3. Graceful Container Self-Healing Circuit

# Verify container failure state before restart
if ! docker inspect --format='{{.State.Health.Status}}' searxng | grep -q "healthy"; then
    echo "[AUTO-HEALER] SearXNG unhealthy. Restarting..."
    docker restart searxng
fi

Best Practices

1. Governed Recovery: Always verify container failure traceback before executing restart operations.
2. Telemetry Log Receipt: Record all auto-healing actions in telemetry_audit.log.

Section 3: Docker Entropy — Thermodynamic Fleet Health

Step 1: Read Fleet Entropy

Run these in order before any intervention:

1. docker ps — what is running, dead, or restarting?
2. docker stats --no-stream — who is consuming CPU/memory?
3. docker system df — how much disk entropy (images, volumes, build cache) has accumulated?
4. docker compose ps and docker compose config if a compose stack is involved.

Step 2: Interpret Signal Priority

Rank findings in this order:

1. Container health status (Up vs Restarting vs Exited)
2. Memory pressure (approaching limit)
3. Disk entropy (dangling images, volumes, build cache)
4. CPU saturation (sustained >80%)
5. Log anomalies (error rate spikes)

Step 3: Choose the Intervention Class

Safe / reversible — proceed with witness:

• docker compose config (validate, no mutation)
• docker compose ps (state inspection)
• docker logs --tail=N <service> (observation)
• docker compose restart <service> (reversible)
• docker compose up -d <service> (restore known state)

Dangerous / irreversible — 888 HOLD required:

• docker system prune -af --volumes (irreversible disk purge)
• docker rm -f <container> (data loss risk)
• docker volume rm <volume> (permanent data destruction)
• docker rmi <image> when the image is actively used

Step 4: Apply Uncertainty Protocol

• Container is Restarting → read logs before restarting again.
• Disk usage >85% → diagnose before pruning.
• Unsure whether a volume is in use → inspect; do not delete.
• Service unhealthy but cause unclear → 888 HOLD.
• Never prune without first running docker system df.

Alert Conditions

Allowed Tools

Forbidden Actions

• NEVER run docker system prune -af as a first response to any problem (panic pruning).
• NEVER restart a container without reading its logs first.
• NEVER assume "container is Up" means "container is healthy".
• NEVER ignore disk entropy until it causes system failure.
• NEVER delete volumes without verifying they contain no critical data.
• NEVER run prune without human confirmation (F1 violation).
• NEVER restart Vault999 — append-only ledger, human ack required.
• NEVER restart multiple containers simultaneously — avoid federation cascade.
• Escalate to arifOS 888_JUDGE before any destructive action.

Output Format

## Skill Result: federation-health

### Summary
One-paragraph summary of fleet thermodynamic state and chosen intervention.

### Evidence
- Container state: <running / restarting / exited>
- Resource pressure: <CPU / memory / disk findings>
- Disk entropy: <docker system df output>
- Log anomalies: <yes/no with snippet>
- Caddy errors: <502/504 count in last N requests>

### Recommendations
- Safe reversible action, OR
- 888 HOLD with reason and proposed next step

### Escalations
- None / <list>

Escalation Path

Consolidated 2026-08-26 from: FORGE-federation-orchestrator, vps-telemetry-auto-healer, FORGE-docker-entropy. AAA Skill Library — version 2.0.0