Red Hat Enterprise Linux 9 — Monitoring and Logging
Companion skill to rhel-server-admin. Covers metrics collection, visualization, log aggregation, alerting, and health checks on RHEL 9.x (and compatible: AlmaLinux 9, Rocky Linux 9, Oracle Linux 9).
Reference Files
Detailed code examples, patterns, and configuration are in the reference files below. Read the relevant file when working on that area.
| File |
Covers |
| loki-pcp-elk-rsyslog-healthchecks.md |
Loki + Promtail, Performance Co-Pilot (PCP), ELK Stack, rsyslog, Cockpit monitoring, health check patterns, and SELinux contexts |
| prometheus-grafana-alertmanager.md |
Prometheus installation and configuration, node_exporter, Grafana dashboards, and Alertmanager setup |
Anti-Patterns
| Anti-Pattern |
Why It Fails |
Correct Approach |
| Alerting on every metric threshold without tuning |
Alert fatigue — operators ignore alerts when 90% are noise; real incidents get buried |
Start with critical alerts only (disk 90%, memory 95%, service down); tune thresholds based on baseline; review alert volume weekly |
| No log rotation or retention policy |
/var/log fills the root filesystem; system stops functioning; log data from 3 years ago wastes storage |
Configure logrotate for all application logs; set journal retention (SystemMaxUse); archive to central logging |
| Monitoring only system metrics, not application health |
CPU and memory look fine while the application returns 500 errors; users report outage before monitoring detects it |
Add application-level health endpoints; monitor HTTP status codes, response latency, and error rates alongside system metrics |
| Installing Prometheus without persistent storage |
Prometheus restart loses all historical data; capacity planning and trend analysis become impossible |
Configure Prometheus with persistent volume (--storage.tsdb.path); set retention period (--storage.tsdb.retention.time) |
| Not correlating logs with metrics during incidents |
Metrics show a spike but no context; logs show errors but no timing; incident diagnosis takes hours instead of minutes |
Use shared labels/tags across metrics (Prometheus) and logs (Loki/ELK); link dashboards to log queries for drill-down |
Related Skills
| Workload |
Skill |
| Core system admin (packages, users, firewall, disk) |
rhel-server-admin |
| Web servers (Nginx, Apache, Caddy) |
rhel-web-servers |
| Databases (PostgreSQL, MySQL, Redis) |
rhel-databases |
| Docker / Podman containers |
rhel-docker-host |
| File sharing (NFS, Samba, Stratis) |
rhel-file-storage |
| DNS, DHCP, NTP |
rhel-network-infra |
| NVIDIA GPU, Ollama, CUDA |
rhel-ollama-nvidia |
1---2name: rhel-monitoring3description: Use when setting up monitoring and logging on RHEL 9 (and AlmaLinux/Rocky 9) — Prometheus and node_exporter, Grafana dashboards, Loki/Promtail log aggregation, ELK stack, Alertmanager, Performance Co-Pilot (PCP), rsyslog, systemd journal management, Cockpit monitoring, and health check patterns. Part of the rhel-* skill family.4---56# Red Hat Enterprise Linux 9 — Monitoring and Logging78Companion skill to `rhel-server-admin`. Covers metrics collection, visualization, log aggregation, alerting, and health checks on RHEL 9.x (and compatible: AlmaLinux 9, Rocky Linux 9, Oracle Linux 9).910<HARD-RULE>11Always verify the RHEL version before applying advice. Packages, paths, and available repos differ between major releases.12```bash13cat /etc/redhat-release14cat /etc/os-release15uname -r16```17</HARD-RULE>1819<HARD-RULE>20Monitoring stacks can consume significant resources. Always size memory and disk before deploying. A Prometheus + Grafana + Loki stack needs at minimum 2 GB RAM and 20 GB disk for small environments. Elasticsearch alone requires at least 4 GB RAM (see JVM heap section).21</HARD-RULE>2223<HARD-RULE>24On RHEL 9, SELinux is enforcing by default. When installing monitoring services that bind to non-standard ports or read log files, you must set proper SELinux contexts and booleans rather than disabling SELinux. Use `ausearch -m AVC --start recent` and `sealert` to diagnose denials.25</HARD-RULE>2627---2829## Reference Files3031Detailed code examples, patterns, and configuration are in the reference files below. Read the relevant file when working on that area.3233| File | Covers |34|---|---|35| [loki-pcp-elk-rsyslog-healthchecks.md](loki-pcp-elk-rsyslog-healthchecks.md) | Loki + Promtail, Performance Co-Pilot (PCP), ELK Stack, rsyslog, Cockpit monitoring, health check patterns, and SELinux contexts |36| [prometheus-grafana-alertmanager.md](prometheus-grafana-alertmanager.md) | Prometheus installation and configuration, node_exporter, Grafana dashboards, and Alertmanager setup |3738---3940---4142## Anti-Patterns4344| Anti-Pattern | Why It Fails | Correct Approach |45|---|---|---|46| Alerting on every metric threshold without tuning | Alert fatigue — operators ignore alerts when 90% are noise; real incidents get buried | Start with critical alerts only (disk 90%, memory 95%, service down); tune thresholds based on baseline; review alert volume weekly |47| No log rotation or retention policy | /var/log fills the root filesystem; system stops functioning; log data from 3 years ago wastes storage | Configure logrotate for all application logs; set journal retention (SystemMaxUse); archive to central logging |48| Monitoring only system metrics, not application health | CPU and memory look fine while the application returns 500 errors; users report outage before monitoring detects it | Add application-level health endpoints; monitor HTTP status codes, response latency, and error rates alongside system metrics |49| Installing Prometheus without persistent storage | Prometheus restart loses all historical data; capacity planning and trend analysis become impossible | Configure Prometheus with persistent volume (--storage.tsdb.path); set retention period (--storage.tsdb.retention.time) |50| Not correlating logs with metrics during incidents | Metrics show a spike but no context; logs show errors but no timing; incident diagnosis takes hours instead of minutes | Use shared labels/tags across metrics (Prometheus) and logs (Loki/ELK); link dashboards to log queries for drill-down |5152---5354## Related Skills5556| Workload | Skill |57|---|---|58| Core system admin (packages, users, firewall, disk) | `rhel-server-admin` |59| Web servers (Nginx, Apache, Caddy) | `rhel-web-servers` |60| Databases (PostgreSQL, MySQL, Redis) | `rhel-databases` |61| Docker / Podman containers | `rhel-docker-host` |62| File sharing (NFS, Samba, Stratis) | `rhel-file-storage` |63| DNS, DHCP, NTP | `rhel-network-infra` |64| NVIDIA GPU, Ollama, CUDA | `rhel-ollama-nvidia` |