# Node Logs

> Retrieve logs from a Kubernetes node — systemd units (journalctl) or files under /var/log. Use when you need node-level evidence: containerd, kubelet, kernel/OOM, or anything the pod's own log cannot show. Three access paths in order: host_script (SSH), node_script (debug pod), and local_script against the kubelet log endpoint when neither is possible.

- Skill: `scitix/node-logs` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add scitix/node-logs`
- Raw SKILL.md: https://api.skillmd.com/api/skills/scitix/node-logs/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: scitix (https://skillmd.com/u/scitix)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/scitix/node-logs

---


# Node Logs

## Read this first: what an empty result means

Every command here can come back with nothing, and the reason matters more than
the emptiness. So both scripts print exactly one `status:` line and it is always
the **last line of the output**, with any explanation above it — one rule, every
outcome. (A bad argument is rejected on stderr with exit 2 before any report, so
there is no status line to read in that case.)

| `status:` | Meaning | What you may conclude |
|-----------|---------|-----------------------|
| `ok` | Source read, lines matched | Read the lines |
| `no_match` | Source read fine, filter matched nothing | **Nothing about the node.** Bounded non-evidence for that one query |
| `source_error` | The log source itself failed (see stderr) | Nothing — the output above it is partial |
| `filter_error` | Your pattern was rejected; nothing was searched | Nothing — fix the pattern and rerun |
| `not_found` / `not_a_log_file` / `empty_source` | The path is wrong, is a directory, or the handler is off | Nothing about the node's health |
| `query_unsupported` | This kubelet cannot serve journald over the API | Nothing — use a file, or tier 1/2 |
| `forbidden` | The agent lacks `nodes/proxy` | Nothing — an access fact, not a node fact |

`no_match` is the one that gets misread. "No PLEG messages in the last hour" is
not "PLEG is healthy": journald retention, a `--since` that misses the event, a
unit that never logged to this source, and a pattern that does not match this
component's wording all produce it. Report what you queried, not just that it
was empty. The header's `scanned:` count is what tells you whether anything was
actually read.

## Pick an access path

Try them in this order and say which one produced the evidence.

### Tier 1 — `host_script` (preferred)

The node is a bound SSH host: check `host_list` by node name or IP. No debug pod,
lowest latency.

**One caveat that produces empty results:** journalctl only shows the *system*
journal to root and to members of `adm` / `systemd-journal`. An ordinary SSH
account gets its own messages and **exit 0** — so a system unit looks like it
logged nothing. The script detects that account state, says so in a `note:`, and
repeats it in the `no_match` explanation. When you see it, the answer is not
"the unit was quiet": rerun via `node_script`, which enters the host namespaces
as root.

```
host_script: host="<host>", skill="node-logs", script="get-node-logs.sh", args="<args>"
```

### Tier 2 — `node_script`

The node is not a bound SSH host. Runs `get-node-logs.sh` inside the host's
namespaces via a privileged debug pod. Same arguments, same output.

```
node_script: node="<node>", skill="node-logs", script="get-node-logs.sh", args="<args>"
```

### Tier 3 — `local_script` against the kubelet log endpoint

Only when tier 1 and tier 2 are both impossible: no SSH binding **and** the debug
pod will not start (the node is NotReady, tainted, out of resources, or the image
cannot be pulled there). This path uses the agent's own kubectl, so it needs
nothing from the node beyond a responsive kubelet.

```
local_script: cluster="<cluster>", skill="node-logs",
              script="get-node-logs-via-api.sh", args="--node <node> --list"
```

**Set `cluster`** to the target cluster's credential name (from `cluster_list`);
`local_script` resolves it to a kubeconfig and exports `KUBECONFIG` for the
script. Do not pass `--kubeconfig` in `args`.

It is a genuinely weaker source, in three specific ways — all three matter when
you report:

- **It serves FILES under `/var/log`, not the journal.** If the node's journald
  does not forward to syslog/messages, kubelet and containerd messages are not
  reachable this way *at all*. A `journal/` entry in the listing means journald
  is persistent, but those files are binary and only `journalctl` can read them —
  which is exactly what tier 1 and 2 give you.
- **No server-side tail, no time filter, no ranges.** A file read transfers the
  whole file. `--head-bytes` can bound it, but it truncates from the *start* of
  the file, i.e. keeps the OLDEST bytes — useless for "what happened 10 minutes
  ago". Prefer a small, specific file over a multi-gigabyte syslog.
- **The journald query path is usually off.** `--query <unit>` uses the kubelet
  node log query (k8s ≥1.30 beta, opt-in alpha in 1.27–1.29). When the feature is
  off the kubelet answers with the `/var/log` **directory listing** and HTTP 200 —
  no error at all. The script detects that and reports `query_unsupported`
  instead of counting HTML as log lines. Always start with `--list`.

## `get-node-logs.sh` (tiers 1 and 2)

Source — one of, repeatable:

| Parameter | Description |
|-----------|-------------|
| `--unit UNIT` | Systemd unit (`containerd`, `kubelet`). Repeat to OR several units in one pass. |
| `--file PATH` | Log file path (`/var/log/messages`). Repeat for several files. |

Time window (journalctl only):

| Parameter | Default | Description |
|-----------|---------|-------------|
| `--since T` | `1h ago` | Window start. Dropped when `--boot` is given, so the boot is not clipped. |
| `--until T` | now | Window end. **Requires an explicit `--since`** — otherwise the default hour would end before it. |
| `--boot ID` | — | `0` = current boot, `-1` = previous, or a boot ID |
| `--list-boots` | — | List boots with IDs and time ranges, then exit |

`T` accepts journalctl syntax (`30m ago`, `today`, `2026-08-18 14:00:00`), RFC3339
(`2026-08-18T06:00:00Z` — converted for you, journalctl rejects that form) or an
epoch (`@1755526800`). The header echoes both what you passed and what was
actually queried.

Filtering — case-insensitive, repeatable, applied in the same single pass:

| Parameter | Description |
|-----------|-------------|
| `--grep PATTERN` | **Extended regular expression.** `pod-a\|pod-b` means a OR b. Repeat for more alternatives. |
| `--grep-fixed STR` | Literal string — no metacharacters. Use for image refs, paths, UUIDs. |

`--grep` and `--grep-fixed` cannot be combined. An invalid ERE is reported as
`filter_error`, never as "nothing found". An **empty** pattern is rejected: it
would match every line and come back as a full hit at `status: ok`, which reads
like a filter that succeeded.

Other:

| Parameter | Default | Description |
|-----------|---------|-------------|
| `--tail N` | `200` | Max log lines printed; matches are counted in full regardless. Clamped to 20000 (the buffer lives on the node) with a `note:` when it clamps. |
| `--include-rotated` | off | With `--file`, also read `PATH.1`, `PATH.2.gz`, … oldest first. Numeric rotations only; name date-suffixed files explicitly. |

Exit codes: `0` ok/no_match · `2` usage or filter error · `3` source unavailable
on this node (no journalctl) · `4` file missing/unreadable · `5` source failed.

### Output

```
source:      journalctl unit=kubelet
window:      since=2026-08-18T06:00:00Z (journalctl: @1755496800)  until=2026-08-18T07:00:00Z (journalctl: @1755500400)
boot:        not restricted (all boots in the window)
filter:      ERE, case-insensitive, 2 pattern(s): pod-a|pod-b
scanned:     14233 line(s) from the source
matched:     17 line(s)
---
Aug 18 06:11:02 node-1 kubelet[1820]: E0818 06:11:02.881 ... pod-a ...
---
status:      ok
```

### Examples

Containerd around an image pull failure — a registry reference is a literal, so
`--grep-fixed` avoids `.` and `-` behaving as regex:

```
host_script: host="node-1", skill="node-logs", script="get-node-logs.sh",
             args='--unit containerd --grep-fixed "myregistry.com/myapp:v1.2" --since "2h ago"'
```

Two pods in one pass over an exact incident window:

```
node_script: node="node-1", skill="node-logs", script="get-node-logs.sh",
             args='--unit kubelet --grep "pod-a|pod-b" --since 2026-08-18T06:00:00Z --until 2026-08-18T07:00:00Z'
```

Kernel OOM evidence across log rotations:

```
node_script: node="node-1", skill="node-logs", script="get-node-logs.sh",
             args='--file /var/log/messages --include-rotated --grep "out of memory|oom-kill"'
```

Did the node reboot, and what did the previous boot end with:

```
host_script: host="node-1", skill="node-logs", script="get-node-logs.sh", args="--list-boots"
host_script: host="node-1", skill="node-logs", script="get-node-logs.sh", args='--unit kubelet --boot -1 --tail 100'
```

## `get-node-logs-via-api.sh` (tier 3)

| Parameter | Description |
|-----------|-------------|
| `--node NODE` | Required |
| `--list [SUBDIR]` | List `/var/log` (or a subdirectory). **Start here** — file names differ per distro: Ubuntu `syslog`, RHEL/CentOS `messages`. |
| `--file NAME` | Read `/var/log/NAME`. Relative path only. |
| `--query UNIT` | Try journald via the kubelet node log query (often unsupported — see above) |
| `--since T` / `--until T` | `--query` only, RFC3339 or epoch. Rejected for file reads, where no time filter exists. |
| `--grep` / `--grep-fixed` | Same semantics as tier 1/2, applied locally |
| `--tail N` | Max log lines printed (default 200, clamped to 20000; in `--query` mode it is also the kubelet's own `tailLines`) |
| `--include-rotated` | Also read `NAME.1`, `NAME.2.gz`, … discovered from one directory listing |
| `--head-bytes N` | Stop each transfer after N bytes — from the file's **oldest** end, and the last line is cut mid-line |

Exit codes: `0` ok/no_match · `2` usage or filter error · `3` not a log file /
query unsupported / handler disabled · `4` file absent · `5` fetch failed ·
`6` forbidden (`nodes/proxy` missing).

```
local_script: cluster="prod", skill="node-logs", script="get-node-logs-via-api.sh",
              args="--node node-1 --list"
local_script: cluster="prod", skill="node-logs", script="get-node-logs-via-api.sh",
              args='--node node-1 --file syslog --grep "kubelet|containerd" --tail 300'
```

## Efficiency: one read, all the questions

The whole pipeline streams once and counts as it goes, in constant memory, so a
long window costs one journalctl read no matter how many patterns you ask about.
Pass every pattern to one invocation (`--grep a --grep b` or `--grep 'a|b'`)
instead of running the script once per pattern — repeating a two-hour journal
read for each keyword is the expensive mistake here, and on tier 3 it re-transfers
the entire file every time.

## Use cases

### Container runtime failures
`--unit containerd` (or `crio`) when pods will not start, images fail to pull, or
containers exit unexpectedly. Match the registry reference with `--grep-fixed`.

### Kubelet problems
`--unit kubelet` for evictions, volume mount failures, PLEG complaints, resource
pressure. Bound the window to the incident with `--since`/`--until` rather than a
wide `2h ago`: a wide window pulls in older, unrelated evictions that are easy to
misattribute to the pod you are investigating.

### Kernel and system events
`--file /var/log/messages` (RHEL) or `/var/log/syslog` (Ubuntu) with
`--include-rotated` for OOM kills, disk errors, and network driver resets. Note
these files only carry what journald forwards to them.

### Node reboots
`--list-boots`, then `--boot -1` to read the end of the previous boot — the
window before an unexpected restart is otherwise easy to miss entirely.

## See also

- `node-health-check` — node conditions, pressure, allocatable vs requested
- `image-pull-debug` — the pull-failure flow that reads containerd through this skill