HestiaCP DevOps AI Skill
Source: webxtekstudio/hestiacp-useful-tools (GitHub, MIT License)
Architecture: Two-brain Orchestrator + Sentinel pattern
Platform: HestiaCP on Debian 12 (Bookworm)
Blueprint: See hestiacp_two_brain_blueprint.md in session artifacts
When to Use
- Server health monitoring and automated reporting
- HestiaCP CLI operations (
v-* commands)
- PHP-FPM pool tuning and troubleshooting (502/504 errors)
- Exim4 mail queue forensics and spam mitigation
- Fail2Ban management and attack response
- SSL certificate management (Let's Encrypt)
- Post-compromise detection (rogue ports, temp executables, webshells)
- Backup verification and disaster recovery
- Nginx/Apache reverse proxy debugging
- CMS permission hardening (WordPress, Laravel, Drupal, Joomla, Magento)
Architecture: Two-Brain Model
Brain 1: Sentinel (Passive Monitor)
- File:
prompts/System-Monitor-Prompt.md
- Input: CRON-injected telemetry (RAM, CPU, Disk, Services, Logs)
- Output: Structured health report (HEALTHY ✅ / WARNING ⚠️ / CRITICAL 🔴)
- Tool:
run_ssh_command (Round 3 deep dives only)
- Key Innovation: Post-compromise detection, attack-is-root-cause principle
Brain 2: Orchestrator (Active DevOps)
- File:
prompts/DevOps-Agent-Prompt.md
- Input: User request or Sentinel alert trigger
- Output: Diagnosis + remediation + verification report
- Tools:
run_ssh_command + ask_knowledge_expert (RAG)
- Constraint: Max 10 ReAct steps
Decision Heuristics (Mandatory)
1. LOOK BEFORE LEAP — cp config.conf config.conf.bak
2. LEAST DESTRUCTIVE — nginx -t BEFORE systemctl restart
3. SILENT EXECUTION — Act safely, report async (no permission prompts in automation)
4. FALSIFIABILITY TEST — After "fixing", run the command that PROVES it's fixed
Noise Filter (False Positive Suppression)
Ignore these — they are NORMAL in HestiaCP:
cloud-init, cloud-config, cloud-final service failures- Apache on port 8080 (backend behind Nginx — this IS the architecture)
- SSH brute-force in auth logs (Fail2Ban handles it)
- High memory if mostly buffers/cache
- Zombie count ≤ 5
- Load spikes during
v-backup-users
- Transient PHP-FPM CPU spikes during backup windows
Severity Classification
| Level |
Trigger |
| ✅ HEALTHY |
Zero thresholds breached, or only marginal |
| ⚠️ WARNING |
Clear threshold breach with operational concern |
| 🔴 CRITICAL |
Service down, disk >90%, backup FAILED, active attack, OOM kills, compromise indicators |
Attack Principle: When PHP-FPM CPU is high AND attack vectors are present, the ROOT CAUSE is the attack — not PHP-FPM. Report as attack, not "busy workload."
Critical Thresholds
| Resource |
Warning |
Critical |
Disk (/ or /home) |
>80% |
>90% |
| Load (1min) |
> nproc |
> nproc × 2 |
| RAM (excl. cache) |
>85% |
— |
| Swap |
>1GB |
>2GB (OOM risk) |
| Mail Queue (Exim) |
>10 messages |
>50 messages |
| Backup Age |
>8 days |
STATUS=FAILED |
| SSL Expiry |
<14 days |
<7 days |
Post-Compromise Detection (6 Vectors)
| Vector |
Check |
Critical |
| Rogue Ports |
ss -tlnp vs expected |
4444, 5555, 6666, 8888 → 🔴 |
| Temp Executables |
/tmp, /var/tmp, /dev/shm |
ANY executable → 🔴 |
| User Audit |
UID ≥ 1000 with /bin/bash |
Unknown user → 🔴 |
| Recent PHP Mods |
Files modified <6hrs |
shell.php, c99.php → 🔴 |
| Outbound C2 |
ss -tnp state established |
IRC 6667, crypto 3333 → 🔴 |
| Suspicious Cron |
curl, wget, base64, /tmp/ |
🔴 possible backdoor |
Service Name Gotchas
| Service |
Correct |
WRONG |
| Mail |
exim4 |
exim |
| Database |
mariadb |
mysql |
| PHP |
php8.2-fpm |
php-fpm |
| DB CLI |
mariadb |
mysql |
PHP-FPM Tuning Formula
Max Total Workers = (Total RAM - 2GB) / 60MB
Low Traffic: 2-5 workers
Medium Traffic: 10-20 workers
High Traffic: 30-50 workers
Exim Log Decoder
<= Email arrived at server
=> Normal delivery (local or external)
-> Forwarded or aliased delivery
** Delivery permanently FAILED (bounced)
== Delivery temporarily DELAYED (queued)
CRITICAL: mainlog does NOT store email subjects. To get subjects, grep the physical Maildir:
/home/<user>/mail/<domain>/<account>/new/ (unread)
/home/<user>/mail/<domain>/<account>/cur/ (read)
False Path Map (Never Use These)
| Generic Path |
HestiaCP Reality |
/etc/letsencrypt/live |
/home/[USER]/conf/web/[DOMAIN]/ssl/ |
/etc/nginx/sites-enabled |
/etc/nginx/conf.d/domains/ |
/var/www/html |
/home/[USER]/web/[DOMAIN]/public_html/ |
Knowledge Base (16 files in knowledge/)
| # |
File |
Size |
Domain |
| 01 |
hestia-system-paths.md |
11.5KB |
Complete path map |
| 02 |
hestia-troubleshooting-logs.md |
5.7KB |
Diagnostic commands |
| 03 |
hestia-cron-guide.md |
2.7KB |
CRON management |
| 04 |
hestia-php-tuning.md |
4.2KB |
PHP-FPM optimization |
| 05 |
hestia-database-guide.md |
2.6KB |
MariaDB config |
| 06 |
hestia-fail2ban-guide.md |
2.0KB |
Jail management |
| 07 |
hestia-ssl-guide.md |
2.2KB |
SSL lifecycle |
| 08 |
hestia-clamav-guide.md |
2.1KB |
Antivirus config |
| 09 |
hestia-exim-troubleshooting.md |
5.0KB |
Mail queue forensics |
| 10 |
hestia-nginx-templates.md |
1.8KB |
Proxy templates |
| 11 |
hestia-backups.md |
3.1KB |
Backup/restore |
| 12 |
hestia-cli-reference.md |
23.2KB |
400+ v-commands |
| 13 |
hestia-common-issues.md |
3.1KB |
Decision trees |
| 14 |
hestia-dovecot-guide.md |
4.4KB |
IMAP/POP3 debug |
| 15 |
exim-forensics.md |
2.8KB |
Maildir forensics |
| 16 |
hestia-custom-tools-extended.md |
6.3KB |
v-fix-web-permissions, v-security-audit |
Integration
- Skill Router: Category 4 (Deploy & Ops)
- Related Skills:
@gcp-cloud-run, @docker-expert, @security-auditor, @systematic-debugging
- MRL Index: Rebuild
mrl_index.pkl to include these 19 files for semantic search
1---2name: hestiacp-devops3description: Autonomous HestiaCP server management with AI agents. Two-brain Orchestrator+Sentinel architecture for VPS monitoring, troubleshooting, security hardening, PHP-FPM tuning, Exim mail forensics, and post-compromise detection. Use when: HestiaCP, VPS, server monitoring, DevOps agent, Linux admin, Nginx/Apache, PHP-FPM, Exim, Fail2Ban, ClamAV, SSL, backup.4---56# HestiaCP DevOps AI Skill78> **Source:** `webxtekstudio/hestiacp-useful-tools` (GitHub, MIT License)9> **Architecture:** Two-brain Orchestrator + Sentinel pattern10> **Platform:** HestiaCP on Debian 12 (Bookworm)11> **Blueprint:** See `hestiacp_two_brain_blueprint.md` in session artifacts1213## When to Use1415- Server health monitoring and automated reporting16- HestiaCP CLI operations (`v-*` commands)17- PHP-FPM pool tuning and troubleshooting (502/504 errors)18- Exim4 mail queue forensics and spam mitigation19- Fail2Ban management and attack response20- SSL certificate management (Let's Encrypt)21- Post-compromise detection (rogue ports, temp executables, webshells)22- Backup verification and disaster recovery23- Nginx/Apache reverse proxy debugging24- CMS permission hardening (WordPress, Laravel, Drupal, Joomla, Magento)2526## Architecture: Two-Brain Model2728### Brain 1: Sentinel (Passive Monitor)29- **File:** `prompts/System-Monitor-Prompt.md`30- **Input:** CRON-injected telemetry (RAM, CPU, Disk, Services, Logs)31- **Output:** Structured health report (HEALTHY ✅ / WARNING ⚠️ / CRITICAL 🔴)32- **Tool:** `run_ssh_command` (Round 3 deep dives only)33- **Key Innovation:** Post-compromise detection, attack-is-root-cause principle3435### Brain 2: Orchestrator (Active DevOps)36- **File:** `prompts/DevOps-Agent-Prompt.md`37- **Input:** User request or Sentinel alert trigger38- **Output:** Diagnosis + remediation + verification report39- **Tools:** `run_ssh_command` + `ask_knowledge_expert` (RAG)40- **Constraint:** Max 10 ReAct steps4142## Decision Heuristics (Mandatory)4344```451. LOOK BEFORE LEAP — cp config.conf config.conf.bak462. LEAST DESTRUCTIVE — nginx -t BEFORE systemctl restart473. SILENT EXECUTION — Act safely, report async (no permission prompts in automation)484. FALSIFIABILITY TEST — After "fixing", run the command that PROVES it's fixed49```5051## Noise Filter (False Positive Suppression)5253Ignore these — they are NORMAL in HestiaCP:54- `cloud-init`, `cloud-config`, `cloud-final` service failures55- Apache on port 8080 (backend behind Nginx — this IS the architecture)56- SSH brute-force in auth logs (Fail2Ban handles it)57- High memory if mostly buffers/cache58- Zombie count ≤ 559- Load spikes during `v-backup-users`60- Transient PHP-FPM CPU spikes during backup windows6162## Severity Classification6364| Level | Trigger |65|---|---|66| ✅ HEALTHY | Zero thresholds breached, or only marginal |67| ⚠️ WARNING | Clear threshold breach with operational concern |68| 🔴 CRITICAL | Service down, disk >90%, backup FAILED, active attack, OOM kills, compromise indicators |6970**Attack Principle:** When PHP-FPM CPU is high AND attack vectors are present, the ROOT CAUSE is the attack — not PHP-FPM. Report as attack, not "busy workload."7172## Critical Thresholds7374| Resource | Warning | Critical |75|---|---|---|76| Disk (`/` or `/home`) | >80% | >90% |77| Load (1min) | > nproc | > nproc × 2 |78| RAM (excl. cache) | >85% | — |79| Swap | >1GB | >2GB (OOM risk) |80| Mail Queue (Exim) | >10 messages | >50 messages |81| Backup Age | >8 days | STATUS=FAILED |82| SSL Expiry | <14 days | <7 days |8384## Post-Compromise Detection (6 Vectors)8586| Vector | Check | Critical |87|---|---|---|88| Rogue Ports | `ss -tlnp` vs expected | 4444, 5555, 6666, 8888 → 🔴 |89| Temp Executables | `/tmp`, `/var/tmp`, `/dev/shm` | ANY executable → 🔴 |90| User Audit | UID ≥ 1000 with `/bin/bash` | Unknown user → 🔴 |91| Recent PHP Mods | Files modified <6hrs | `shell.php`, `c99.php` → 🔴 |92| Outbound C2 | `ss -tnp state established` | IRC 6667, crypto 3333 → 🔴 |93| Suspicious Cron | `curl`, `wget`, `base64`, `/tmp/` | 🔴 possible backdoor |9495## Service Name Gotchas9697| Service | Correct | WRONG |98|---|---|---|99| Mail | `exim4` | ~~exim~~ |100| Database | `mariadb` | ~~mysql~~ |101| PHP | `php8.2-fpm` | ~~php-fpm~~ |102| DB CLI | `mariadb` | ~~mysql~~ |103104## PHP-FPM Tuning Formula105106```107Max Total Workers = (Total RAM - 2GB) / 60MB108109Low Traffic: 2-5 workers110Medium Traffic: 10-20 workers111High Traffic: 30-50 workers112```113114## Exim Log Decoder115116```117<= Email arrived at server118=> Normal delivery (local or external)119-> Forwarded or aliased delivery120** Delivery permanently FAILED (bounced)121== Delivery temporarily DELAYED (queued)122```123124**CRITICAL:** `mainlog` does NOT store email subjects. To get subjects, grep the physical Maildir:125```126/home/<user>/mail/<domain>/<account>/new/ (unread)127/home/<user>/mail/<domain>/<account>/cur/ (read)128```129130## False Path Map (Never Use These)131132| Generic Path | HestiaCP Reality |133|---|---|134| `/etc/letsencrypt/live` | `/home/[USER]/conf/web/[DOMAIN]/ssl/` |135| `/etc/nginx/sites-enabled` | `/etc/nginx/conf.d/domains/` |136| `/var/www/html` | `/home/[USER]/web/[DOMAIN]/public_html/` |137138## Knowledge Base (16 files in `knowledge/`)139140| # | File | Size | Domain |141|---|---|---|---|142| 01 | `hestia-system-paths.md` | 11.5KB | Complete path map |143| 02 | `hestia-troubleshooting-logs.md` | 5.7KB | Diagnostic commands |144| 03 | `hestia-cron-guide.md` | 2.7KB | CRON management |145| 04 | `hestia-php-tuning.md` | 4.2KB | PHP-FPM optimization |146| 05 | `hestia-database-guide.md` | 2.6KB | MariaDB config |147| 06 | `hestia-fail2ban-guide.md` | 2.0KB | Jail management |148| 07 | `hestia-ssl-guide.md` | 2.2KB | SSL lifecycle |149| 08 | `hestia-clamav-guide.md` | 2.1KB | Antivirus config |150| 09 | `hestia-exim-troubleshooting.md` | 5.0KB | Mail queue forensics |151| 10 | `hestia-nginx-templates.md` | 1.8KB | Proxy templates |152| 11 | `hestia-backups.md` | 3.1KB | Backup/restore |153| 12 | `hestia-cli-reference.md` | 23.2KB | **400+ v-commands** |154| 13 | `hestia-common-issues.md` | 3.1KB | Decision trees |155| 14 | `hestia-dovecot-guide.md` | 4.4KB | IMAP/POP3 debug |156| 15 | `exim-forensics.md` | 2.8KB | Maildir forensics |157| 16 | `hestia-custom-tools-extended.md` | 6.3KB | v-fix-web-permissions, v-security-audit |158159## Integration160161- **Skill Router:** Category 4 (Deploy & Ops)162- **Related Skills:** `@gcp-cloud-run`, `@docker-expert`, `@security-auditor`, `@systematic-debugging`163- **MRL Index:** Rebuild `mrl_index.pkl` to include these 19 files for semantic search
Fandry96