# Linux Observability

> Use when adding or diagnosing Prometheus/node_exporter metrics, central log shipping, or lightweight health endpoints; use linux-system-monitoring for an ad hoc host snapshot and linux-log-management for local log analysis.

- Skill: `peterbamuhigire/linux-observability` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add peterbamuhigire/linux-observability`
- Raw SKILL.md: https://api.skillmd.com/api/skills/peterbamuhigire/linux-observability/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: peterbamuhigire (https://skillmd.com/u/peterbamuhigire)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/peterbamuhigire/linux-observability

---


# Linux Observability

## Distro support

Prometheus, `node_exporter`, and Grafana are **portable** (static binaries /
cross-platform packages); the systemd units are identical. The differences are
install source and opening the scrape port on the firewall. Body uses
Debian/Ubuntu; the **RHEL family** (Fedora, RHEL, CentOS Stream, Rocky, Alma,
Oracle) equivalents are in the matrix.

| Concept | Debian/Ubuntu | RHEL family |
|---|---|---|
| Install node_exporter | apt / release binary | dnf (EPEL has `golang-github-prometheus-node-exporter`) / release binary |
| Open scrape port (9100) | `ufw allow 9100/tcp` | `firewall-cmd --add-port=9100/tcp --permanent && firewall-cmd --reload` |
| Install Telegraf | InfluxData apt repo / `apt install telegraf` | InfluxData yum repo / `dnf install telegraf` |
| Install Datadog agent | official script / Datadog apt repo | official script / Datadog dnf repo |
| Open Telegraf scrape port (9273) | `ufw allow 9273/tcp` (restrict to monitor IP) | `firewall-cmd --add-rich-rule ... port=9273` |
| systemd unit / service | identical | identical |
| Log forwarding | rsyslog / journald | same (journald primary on RHEL) |
| Health endpoint pattern | identical | identical |

**RHEL-family note:** `node_exporter` is in **EPEL** on RHEL/Rocky/Alma
(`ensure_epel`), in main on Fedora, or install the upstream static binary on
either. Open the firewall with firewalld, not ufw.

In `sk-*` scripts use the `common.sh` primitives (`pkg_install`, `ensure_epel`,
`firewall_allow`) instead of hardcoding the family. See
[`linux-bash-scripting`](../../10-automation-and-scripting/linux-bash-scripting/SKILL.md) and
[`docs/multi-distro/plan.md`](../../docs/multi-distro/plan.md).

## Use when

- Adding metrics collection, log shipping, or health endpoints to a server.
- Making a host visible to Prometheus or a central logging pipeline.
- Standardizing `/health` behavior for operational checks.

## Do not use when

- The task is reading local logs only; use `linux-log-management`.
- The task is general performance triage without telemetry design work; use `linux-system-monitoring`.

<!-- dual-compat-start -->

## Required Inputs

| Artefact | Source | Required? | If absent |
|---|---|---|---|
| Telemetry objective and service | Service owner/SLO | yes | Stop before installing an unfocused collector |
| Collector or log destination contract | Platform owner | yes | Produce design only; do not expose a port or ship logs |
| Network, authentication, labels, retention | Platform/security policy | for integration | Mark end-to-end delivery unassessed |
| Change authority and rollback window | Operations owner | for mutation | Limit to read/search inspection and proposed config |

## Capability Contract

Read/search access supports current-state assessment. Installing exporters/agents, editing units/config, opening ports, transmitting logs, or reloading services requires explicit authority. Use least-privilege credentials and never expose sensitive health data.

## Degraded Mode

Fallback when receiver access is unavailable: validate only the reachable segment and label end-to-end scrape/delivery unassessed. Apply the same limit when network, credentials, root, or binaries are missing; a local listener is not proof of monitoring.

## Decision Rules

| Choice | Action | Failure avoided |
|---|---|---|
| Prometheus can pull securely | Use scrape/exporter model | Unnecessary push state |
| Logs leave the host | Define filtering, TLS, backpressure, and retention | Data leak or disk exhaustion |
| Health dependency is slow/optional | Keep liveness cheap; separate readiness | Restart storms |
| Scrape port is required | Restrict source to collector | Public telemetry exposure |

## Workflow

1. Define the operational question, consumer, labels, retention, and access boundary.
2. Inspect existing collectors, ports, units, and pipelines before adding components.
3. Choose metrics, logs, or health semantics using the decision table; stop on undefined sensitive fields.
4. Back up config, validate syntax, and apply the smallest authorised change.
5. Verify from the consumer side, including auth, timestamps, labels, and failure state.
6. On failure, recover prior config/service state and report which segment remains unassessed.

## Evidence Produced

| Artefact | Acceptance |
|---|---|
| Consumer-side telemetry proof | Shows target, timestamp, expected metric/log/health value, and labels |
| Configuration evidence | Contains redacted diff, syntax check, service status, and rollback |
| Coverage statement | Names monitored components, failure states, retention, and unassessed segments |

## Quality standards

- Observability changes must be testable from the receiving system, not only from the host.
- Keep health endpoints cheap, deterministic, and safe to expose.
- Prefer standard, maintainable telemetry patterns over bespoke one-offs.

## Anti-patterns

- Declaring success from a local listener. Fix: verify scrape or receipt at the consumer.
- Shipping logs without destination/retention. Fix: define transport, filtering, backpressure, and expiry.
- Making `/health` a heavy diagnostic. Fix: keep liveness cheap and separate readiness/dependency detail.
- Opening exporter ports globally. Fix: restrict sources to approved collectors.
- Putting secrets or high-cardinality values in labels. Fix: use bounded non-sensitive dimensions.
- Ignoring pipeline failure behaviour. Fix: test receiver outage, buffering, and recovery.

## Outputs

| Artefact | Consumer | Acceptance condition |
|---|---|---|
| Telemetry integration | Operations | Consumer receives the intended signal with correct labels/timestamps |
| Operational runbook note | On-call | Names failure signal, retention, access, and recovery steps |
| Exposure record | Security owner | Documents listener, allowed sources, auth/TLS, and residual risk |

## Worked Example

For node_exporter on Rocky Linux, restrict TCP 9100 to the Prometheus source, validate the unit, then prove the target is `UP` and `node_uname_info` carries the expected instance label. A successful local `curl` alone leaves consumer delivery unassessed.

<!-- dual-compat-end -->

## References

- [`../../docs/continuous-improvement/value-stream-5s-qc-story.md`](../../docs/continuous-improvement/value-stream-5s-qc-story.md)
- [`../../docs/continuous-improvement/incident-learning-standard.md`](../../docs/continuous-improvement/incident-learning-standard.md)

- [`references/prometheus-setup.md`](references/prometheus-setup.md)
- [`references/telemetry-agents.md`](references/telemetry-agents.md)
- [`references/log-forwarding.md`](references/log-forwarding.md)
- [`references/health-endpoint-pattern.md`](references/health-endpoint-pattern.md)

**This skill is self-contained.** Every step below uses standard tools and
released binaries; the body shows Debian/Ubuntu, with RHEL-family equivalents
in the **Distro support** matrix above. The `sk-*` scripts in the **Optional
fast path** section are convenience wrappers — never required.

This skill owns **metrics and log shipping** — turning a server from a
black box you SSH into occasionally into a first-class citizen of a
monitoring stack.

Complements `linux-system-monitoring` (local, interactive: `top`, `htop`,
`iostat`) with remote, continuous, aggregated observability (Prometheus
scrape targets, Loki log streams, standard `/health` endpoints).

Informed by *Linux System Administration for the 2020s* (observability as
mandatory, not optional).

---

## When to use

- Installing Prometheus `node_exporter` on a server.
- Creating a standard `/health` endpoint for a web service.
- Forwarding logs from a server to a central collector (Loki, ELK,
  CloudWatch).
- Auditing which metrics and logs a server is currently emitting.

## When NOT to use

- `top`, `htop`, `iostat` for right-now triage → `linux-system-monitoring`.
- Grepping local log files → `linux-log-management`.
- Prometheus/Grafana *alerting rules* — those live in the monitoring
  server's config, not on the managed host.

---

## Standing rules

1. **Every managed server exposes `node_exporter` on `127.0.0.1:9100`.**
   UFW allows scrape only from the monitoring server's IP — never from
   `0.0.0.0`. Exporter runs as its own unprivileged user.
2. **Every web service has a `/health` endpoint** that checks at minimum:
   database reachable, disk not full, required services running. 200 on
   healthy, 503 on unhealthy, JSON body with per-check status.
3. **Logs ship off the server.** Local files are for operator grep, not
   long-term storage.
4. **Metrics have cardinality discipline.** No per-request unique IDs as
   labels. Ever. Prometheus explodes on high cardinality.
5. **Public vs internal health endpoints are split.** Public `/health`
   returns minimal info (200/503); internal `/health/detail` (behind
   firewall or auth) returns per-check JSON.
6. **TLS on the wire for log shipping.** Always. No plain-text logs
   crossing the network — they contain PII and tokens.

---

## Quick reference — manual commands

### Install node_exporter from GitHub release (canonical method)

```bash
# 1. Download and verify (check releases: https://github.com/prometheus/node_exporter)
VER=1.8.2
cd /tmp
curl -LO https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz
curl -LO https://github.com/prometheus/node_exporter/releases/download/v${VER}/sha256sums.txt
sha256sum -c sha256sums.txt --ignore-missing 2>&1 | grep OK

# 2. Install
tar xzf node_exporter-${VER}.linux-amd64.tar.gz
sudo install -m 0755 node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/

# 3. Unprivileged user
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exp

# 4. Systemd unit
sudo tee /etc/systemd/system/node_exporter.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus node exporter
After=network-online.target
Wants=network-online.target

[Service]
User=node_exp
Group=node_exp
ExecStart=/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9100
Restart=on-failure
RestartSec=3
ProtectSystem=strict
ProtectHome=yes
NoNewPrivileges=yes
CapabilityBoundingSet=
PrivateTmp=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter --no-pager

# 5. UFW — allow only from monitoring host
sudo ufw allow from 10.0.0.5 to any port 9100 proto tcp

# 6. Verify
curl -s localhost:9100/metrics | head -20
```

Full deep-dive (collectors, textfile collector, other exporters,
systemd hardening options, cardinality discipline) — see
[`references/prometheus-setup.md`](references/prometheus-setup.md).

### Other telemetry agents (Telegraf, Datadog) — alternatives to node_exporter

`node_exporter` is the **recommended default**: OSS, pull-based, minimal attack
surface, scraped by your own Prometheus. Two alternatives exist when you have a
concrete reason to use them:

| Agent | Model | Use when |
|---|---|---|
| **node_exporter** (default) | OSS, pull | You run (or want) your own Prometheus/Grafana stack — keeps the smallest surface. |
| **Telegraf** (InfluxData) | OSS, push | You already run InfluxDB, or want one plugin-rich agent (systemd/SNMP/MySQL inputs) pushing to InfluxDB or a Prometheus `/metrics` endpoint. Stays inside your network. |
| **Datadog agent** | **SaaS, push** | Your org pays for Datadog and wants metrics+logs+APM in one managed platform — **and accepts that operational data (and logs, which carry PII/secrets) leaves your network** to Datadog's cloud. |

Telegraf install (both families, signed repo), inputs
(cpu/mem/disk/net/systemd) + outputs (InfluxDB / Prometheus), Datadog install
via the official script/repo, `datadog.yaml` API-key config, enabling
integrations, and the **privacy/egress** consideration of a SaaS agent — all in
[`references/telemetry-agents.md`](references/telemetry-agents.md).

**API keys / tokens** (Telegraf InfluxDB token, Datadog API key) are never put
in a world-readable config: store them in a `0600`/root systemd env file and
inject via the unit — see [`linux-secrets`](../../02-users-access-and-secrets/linux-secrets/SKILL.md).

### Log forwarding with rsyslog (TLS)

```bash
sudo apt install rsyslog-gnutls
sudo mkdir -p /etc/rsyslog.d

# Minimal forwarder config (rsyslog.d/90-forward.conf)
sudo tee /etc/rsyslog.d/90-forward.conf >/dev/null <<'EOF'
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/ssl/certs/loghost-ca.crt
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer loghost.internal

$ActionQueueType LinkedList
$ActionQueueFileName fwdRule1
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on

*.* @@loghost.internal:6514
EOF

sudo systemctl restart rsyslog
```

Full setup with fluent-bit, vector, promtail, and journald forwarding —
see [`references/log-forwarding.md`](references/log-forwarding.md).

### Minimal `/health` endpoint (Nginx → PHP)

```bash
# A /health.php file in your web root:
sudo tee /var/www/html/health.php >/dev/null <<'PHP'
<?php
header('Content-Type: application/json');
$checks = [];
$ok = true;

// DB reachable
try {
    $pdo = new PDO('mysql:host=127.0.0.1;dbname=app', 'health', getenv('HEALTH_PASS'));
    $pdo->query('SELECT 1');
    $checks['db'] = 'ok';
} catch (Throwable $e) {
    $checks['db'] = 'fail';
    $ok = false;
}

// Disk < 90% on / and /var
foreach (['/', '/var'] as $mount) {
    $pct = 100 - (disk_free_space($mount) / disk_total_space($mount) * 100);
    $checks["disk:$mount"] = $pct < 90 ? 'ok' : 'fail';
    if ($pct >= 90) $ok = false;
}

http_response_code($ok ? 200 : 503);
echo json_encode(['status' => $ok ? 'ok' : 'fail', 'checks' => $checks]);
PHP
```

Full pattern (Nginx-only health via `stub_status`, Node.js variant,
caching, auth for detail endpoint) — see
[`references/health-endpoint-pattern.md`](references/health-endpoint-pattern.md).

---

## Typical workflows

### Workflow: "Make this server visible to Prometheus"

1. Install node_exporter (command sequence above).
2. `sudo ufw allow from <monitor-ip> to any port 9100 proto tcp`.
3. `curl -s localhost:9100/metrics | head -20` — confirm metrics.
4. Add the server as a scrape target on your Prometheus server (outside
   the scope of this skill — that lives on the monitoring host).

### Workflow: "Add /health to an existing vhost"

1. Write the `health.php` (or Node.js equivalent) above.
2. In the Nginx vhost, add:
   ```nginx
   location = /health {
       include fastcgi_params;
       fastcgi_pass unix:/run/php/php8.3-fpm.sock;
       fastcgi_param SCRIPT_FILENAME /var/www/html/health.php;
       access_log off;
   }
   ```
3. `sudo nginx -t && sudo systemctl reload nginx`.
4. Test: `curl -sI https://example.com/health` → 200 or 503.

### Workflow: "Forward logs to a central Loki collector"

1. Install fluent-bit (`apt install fluent-bit` or upstream repo for
   newer versions).
2. Drop `/etc/fluent-bit/fluent-bit.conf` from
   [`references/log-forwarding.md`](references/log-forwarding.md).
3. `sudo systemctl enable --now fluent-bit`.
4. Verify arrival at the Loki server.

---

## Troubleshooting / gotchas

- **`node_exporter` listening on 0.0.0.0 by accident.** The systemd unit
  above pins `127.0.0.1:9100` — verify with `ss -tlnp | grep 9100`. If
  it's bound everywhere, fix the `--web.listen-address` flag and reload.
- **UFW allows the scrape but Prometheus can't reach it.** Check that
  the rule is `from <monitoring-ip>` not `to <monitoring-ip>`. Easy
  direction confusion.
- **High-cardinality metric explodes Prometheus.** Signs: Prometheus OOMs
  after adding a new exporter. Usually a label with request IDs or user
  IDs. Drop the offending metric or relabel it away.
- **`/health` page cached by CDN.** Set `Cache-Control: no-store` in the
  Nginx location. A cached 200 will hide a real 503 from the load
  balancer for minutes.
- **rsyslog forwarding queues on disk fill /var/spool.** Set a max queue
  size on the `$ActionQueue*` config, and monitor `/var/spool/rsyslog`.
- **Fluent-bit does not automatically rotate its own logs.** Configure
  its log target carefully — it's a log shipper that can create logs.

---

## References

- [`references/prometheus-setup.md`](references/prometheus-setup.md) —
  full node_exporter install, other exporters, cardinality discipline,
  systemd hardening.
- [`references/telemetry-agents.md`](references/telemetry-agents.md) —
  Telegraf (InfluxData) and Datadog agent as alternatives to node_exporter:
  install on both families, Telegraf inputs/outputs (InfluxDB/Prometheus),
  Datadog `datadog.yaml`/API key/integrations, "when to use which", and the
  privacy/egress consideration of a SaaS agent.
- [`references/log-forwarding.md`](references/log-forwarding.md) —
  rsyslog/fluent-bit/vector/promtail with TLS forwarding examples.
- [`references/health-endpoint-pattern.md`](references/health-endpoint-pattern.md) —
  full `/health` pattern with PHP and Node.js examples, internal detail
  endpoint, caching strategy.
- Book: *Linux System Administration for the 2020s* — observability is
  mandatory, not optional.
- Man pages: `rsyslogd(8)`, `systemd.service(5)`.

---

## Optional fast path (when sk-* scripts are installed)

Running `sudo install-skills-bin linux-observability` installs:

| Task | Fast-path script |
|---|---|
| Install node_exporter as unprivileged systemd service | `sudo sk-node-exporter-install --monitor-ip <ip>` |
| Install Telegraf (OSS alternative) with metrics inputs + an output | `sudo sk-telegraf-setup --output prometheus` |
| Create/verify a `/health` endpoint for a vhost | `sudo sk-health-endpoint --domain <d> --db mysql` |
| Configure log forwarding over TLS | `sudo sk-log-forward-setup --collector <host>:<port> --tls` |

These are optional wrappers around the manual steps above.

## Scripts

This skill installs the following scripts to `/usr/local/bin/`. To install:

```bash
sudo install-skills-bin linux-observability
```

| Script | Source | Core? | Purpose |
|---|---|---|---|
| sk-node-exporter-install | scripts/sk-node-exporter-install.sh | no | Install Prometheus node_exporter as unprivileged systemd service, firewall-restrict, verify scrape. |
| sk-telegraf-setup | scripts/sk-telegraf-setup.sh | no | Install InfluxData Telegraf (OSS alternative to node_exporter) on both families from the signed repo; write host-metrics inputs (cpu/mem/disk/net/systemd) + one output (Prometheus `/metrics` or InfluxDB v2); firewall-restrict the scrape port; validate with `telegraf --test`. Token read from the 0600 env file, never the config. |
| sk-health-endpoint | scripts/sk-health-endpoint.sh | no | Create and verify `/health` for a vhost: checks db, disk, required services; 200/503 + JSON. |
| sk-log-forward-setup | scripts/sk-log-forward-setup.sh | no | Configure rsyslog or fluent-bit to forward journald and webserver logs to a central collector over TLS. |