delorenj Monitoring Stack
Stack root: ~/docker/stacks/monitoring/
Architecture
See references/architecture.md for full service map, ports, URLs, and data flow.
Common Operations
Restart the full stack
cd ~/docker/stacks/monitoring && docker compose up -d
Restart a single service
cd ~/docker/stacks/monitoring && docker compose up -d <service-name>
Hot-reload Prometheus rules (no restart needed)
curl -s -X POST http://localhost:9472/-/reload
Hot-reload Alertmanager config (no restart needed)
curl -s -X POST http://localhost:9784/-/reload
Verify process-exporter is scraping
curl -s http://localhost:9256/metrics | grep namedprocess | head -5
Test Telegram alert delivery
import urllib.request, json
url = "https://api.telegram.org/bot<TOKEN>/sendMessage"
data = json.dumps({"chat_id": 7564050286, "text": "Test alert", "parse_mode": "HTML"}).encode()
req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"})
urllib.request.urlopen(req)
Bot token is in ~/docker/stacks/monitoring/alertmanager/config.yml.
Adding New Alert Rules
See references/alert-rules-guide.md for rule file locations, PromQL patterns, severity conventions, and examples.
Quick path: Edit the appropriate rule file, then hot-reload Prometheus:
curl -s -X POST http://localhost:9472/-/reload
Rule file locations
| File | Scope |
|---|---|
prometheus/alert_rules.yml |
Container CPU, memory, availability |
prometheus/system_alerts.yml |
Host system + per-process hogs |
prometheus/rules/docker-health.yml |
Container health checks, restart loops |
Alerting
Alerts route to Telegram via @DeLoNETBot (chat_id: 7564050286).
Config: alertmanager/config.yml
Severity routing:
critical: repeat every 1hwarning: repeat every 4h
Key Design Decisions
- cadvisor is resource-capped: 1 CPU, 512M memory, 30s housekeeping interval, pruned metrics. Without these limits it will eat an entire core scanning 69+ containers.
- process-exporter exists specifically to catch host-level hogs that cadvisor and node-exporter miss (per-process CPU/memory).
- Duplicate rules were consolidated:
docker-health.ymlonly has health-check and restart-loop rules. CPU/memory rules live inalert_rules.yml.
Diagnosing Performance Issues
When the system stutters, check in this order:
ps aux --sort=-%cpu | head -15- find CPU hogsfree -h- check swap pressuresensors- check thermals (Tctl)docker stats --no-stream- find container hogs
The alert rules should catch most of these automatically now. If they don't fire, check:
- Prometheus is up:
curl http://localhost:9472/-/healthy - Alertmanager is up:
curl http://localhost:9784/-/healthy - Rules loaded:
curl http://localhost:9472/api/v1/rules | python3 -m json.tool | head -40