stern
stern tails logs from many pods and many containers at once, selected by a regex or a
Kubernetes resource, with automatic pickup of new pods and colorized per-pod output.
Iron Law
In an agent/non-interactive context, every stern invocation carries --no-follow.
Without it stern streams forever and never exits — the tool call blocks until the harness
timeout, and the partial output may be discarded. The only exception: the user explicitly asked to
watch a live stream, and it runs as a background command with a stated stop condition.
The second half of the law: bound the volume. The defaults are --since 48h and --tail -1
(every line ever retained), multiplied by every matching pod. Always narrow at least one of
--tail / --since — and both once the query spans more than a handful of containers, since
they are what truncates each stream. Narrowing which containers are read at all (-E,
--exclude-pod, -l, a tighter query) saves more and costs nothing; line filters save neither.
Rationalization table
Every one of these will occur to you mid-task. All of them are wrong.
| The thought |
Reality |
| "This namespace only has two pods, streaming is fine." |
Pod count does not make stern exit. Two pods stream forever just as well as fifty. |
| "The user said 'watch'/'monitor'/'看一下現在的狀況', so they want live output." |
They want an answer. --no-follow --since 5m gives it. Streaming into a blocked tool call gives them nothing. |
| "I'll add a timeout, that bounds it." |
timeout exits 124 and can cut a line mid-write, so the harness may surface a failed command instead of your logs. Bound it with stern's own flags; keep timeout only as a belt-and-braces outer limit. |
| "It's idle, it will stop when there's nothing left." |
--no-follow is the only thing that makes stern conclude "all logs shown". Idle means it waits, not exits. |
"I'll pipe to head, so it ends." |
It does end, but head keeps the oldest N lines it happened to see first — the opposite of the recent lines you wanted. --tail N selects from the end. |
| "The previous stern call worked without it." |
It did not "work" — it hit the timeout, and you saw the partial output that survived. |
"This is a follow-up to a live session, --tail 0 is what they meant." |
--tail 0 means "only lines from now on", which by definition never terminates. It is also rejected together with --no-follow. |
The load-bearing test is not "is streaming reasonable here?" but "who reads the output?" If the
answer is you, in this tool call, then --no-follow.
The default command shape
stern <query> -n <namespace> --no-follow --tail 50 --since 1h --color never --only-log-lines
| part |
why |
--no-follow |
terminates; without it the call hangs |
--tail 50 |
last N lines per container — raise deliberately, not by default |
--since 1h |
overrides the 48h default |
--color never |
no ANSI escapes to confuse parsing/diffing |
--only-log-lines |
drops the + pod ... / - pod ... attach/detach status lines |
Keep the status lines (i.e. omit --only-log-lines) when you need to know which pods were even
found — that is often the actual answer.
If the task already has a familiar shape — a crash loop, hunting one error across a namespace,
getting output in chronological order, replaying a local log file — take the worked command from
references/recipes.md instead of composing flags from scratch.
Selecting what to read
The positional argument is a regex on pod names, unless it has the form <resource>/<name>,
which is an exact resource lookup:
stern . --no-follow # every pod in the namespace (regex "." matches all)
stern 'web-\w' --no-follow # web-backend, web-frontend — but not web-123
stern deploy/nginx --no-follow # all pods belonging to that Deployment
stern svc/api --no-follow # all pods behind that Service
Supported resources (with aliases): pod/po, replicationcontroller/rc, service/svc,
daemonset/ds, deployment/deploy, replicaset/rs, statefulset/sts, job.
Plural forms work too (deployments/nginx).
Narrow further with:
| need |
flag |
| one container in multi-container pods |
-c <regex> |
| drop a sidecar (istio, linkerd…) |
-E istio-proxy |
| drop noisy pods |
--exclude-pod <regex> |
| by label |
-l app=api |
| by field |
--field-selector spec.nodeName=node-1 |
| by node |
--node node-1 |
| several namespaces |
-n a,b (repeatable) / -A for all |
| only containers that have already exited (finished Jobs) |
--container-state terminated |
| skip init/ephemeral containers |
--init-containers=false, --ephemeral-containers=false — on a mesh-injected pod this also drops the sidecar, see below |
Since Kubernetes 1.28 a native sidecar is an init container with restartPolicy: Always, so
--init-containers=false on a mesh-injected pod removes the proxy too. That is usually what you
want — but the proxy's access logs carry x-request-id, which is what makes following one request
across services possible. Drop istio-init alone with -E istio-init when you need to keep it.
That table is the working subset. When you need a flag that is not in it, an exact default, or the
config file that may silently be changing those defaults, read
references/flags.md — guessing a flag name costs a failed invocation.
--include / -i and --exclude / -e filter log lines (regex, repeatable) — prefer them over
piping to grep, because they apply before the lines are formatted. They do not make the query
cheaper: filtering happens locally on lines already received. To spend less, cut containers with the
selection flags above, or truncate with --tail/--since.
stern deploy/api --no-follow --tail 200 -i 'ERROR|panic' -e 'health.?check'
Reading the result correctly
Machine-readable output
# no pipe: stern's own exit status is the answer
stern deploy/api --no-follow --tail 100 -o json --only-log-lines
# piped: pipefail belongs in the command, not in a footnote.
# -i '^\s*\{' keeps plain-text framework lines away from jq
set -o pipefail
stern deploy/api --no-follow --tail 100 -o raw --only-log-lines -i '^\s*\{' | jq
Three things in that snippet, each preventing a different wrong answer:
--only-log-lines, not a redirect. The + pod › container attach lines go to stderr, so
2>&1 | jq feeds jq a non-JSON line and the pipeline aborts to nothing; 2>/dev/null avoids
that but throws away stern's real errors (RBAC forbidden, a bad --context) along with the
noise. --only-log-lines stops the status lines being printed at all and leaves errors on stderr.
set -o pipefail. jq exits 0 on empty input, so without it the pipeline reports success even
when stern failed. It is portable — prefer it. Where you cannot set it, check the first command's
status rather than $?: ${PIPESTATUS[0]} in bash, ${pipestatus[1]} in zsh (lowercase, and
1-indexed). Using the bash spelling under zsh expands to an empty string, which reads as "did not
fail" — the exact failure this bullet exists to prevent, on the shell macOS defaults to.
-i '^\s*\{' with -o raw. -o raw passes the application's own line through, and real
services mix formats — framework and stdlib lines are plain text alongside the JSON logger. jq
aborts on the first one, prints nothing and exits 5, which pipefail turns into a failed pipeline
for perfectly healthy logs. -o json needs no guard: that envelope is always valid JSON. Keep the
\s* — a bare ^\{ drops an indented record silently at exit 0. Guarding on the jq side instead
takes two filters, jq -R 'fromjson? | objects | …': ? guards only fromjson, so without
objects a top-level array or scalar parses, reaches the field filter and raises a per-record
error that pipefail cannot see — jq reports only the last input's status.
For the -o json field names, the other predefined outputs, and --template for reshaping JSON
application logs into something readable, see references/templates.md.
Preflight
command -v stern — if missing, brew install stern (macOS/Linux) or
kubectl krew install stern; kubectl logs is the fallback for a single pod.
- Know which cluster you are about to read:
kubectl config current-context. Confirm with the user
before pulling logs from an unfamiliar or production-looking context, and remember that log
content may contain secrets — quote selectively rather than dumping it wholesale.
- stern honours
$KUBECONFIG; --kubeconfig and --context override it.
Gotchas
| symptom |
cause / fix |
| command hangs |
missing --no-follow |
Error: --no-follow cannot be used with --tail=0 |
--tail=0 means "only new lines", which contradicts exiting; use --tail 1 or drop --no-follow |
--condition rejected |
it is only supported with --tail=0 or --no-follow |
| flood of output |
--since 48h + --tail -1 defaults across many pods |
| no logs from a crash-looping pod |
not a state-filter problem: the default --container-state all already covers it, and stern falls back to the last terminated instance's logs. Check --since/--tail first. --container-state terminated would exclude it — CrashLoopBackOff is waiting |
| a container is skipped entirely |
it has no container ID yet (never started — image pull failure, etc.); its logs do not exist, use kubectl describe pod |
a wide query takes ~30s with no error, or logs "client-side throttling" |
client-go's default rate limiter (QPS 5, burst 10), not stern or the cluster. --max-log-requests does not affect it — add --qps=-1, or --qps 50 --burst 100 on a shared cluster. At default concurrency the wait is silent, so assume throttling before blaming the API server |
still slow after --qps=-1 |
transfer volume, not rate limiting. --include/--exclude filter client-side, so a filtered query reads exactly as much as an unfiltered one. Drop whole containers first (-E, --exclude-pod, -l, a tighter query), then truncate what remains with --tail/--since |
| ANSI garbage in captured output |
--color never |
--timestamps short gives the long format |
the = cannot be omitted. --timestamps short silently ignores the value and falls back to the full format — no error. Write --timestamps=short, or bare -t |
parseJSON template suddenly hits its else branch |
-t is on: it prefixes the timestamp into .Message. Drop -t when parsing JSON |
empty result from stern … | anything |
either 2>&1 merged the stderr status lines into the pipe and the consumer aborted, or stern failed and the consumer returned 0 anyway — jq, sort, grep -c, wc all exit 0 on empty input. Use --only-log-lines and set -o pipefail on every stern pipeline; never 2>/dev/null, which hides the error that explains it |
| running inside a Pod: forbidden |
needs RBAC get,watch,list on pods and pods/log |
Each reference is linked above from the point where it becomes the right thing to read:
recipes.md for worked commands, flags.md for the
complete flag set and config file, templates.md for output formatting.
1---2name: stern-logs3description: Read, filter and search Kubernetes pod logs with stern — the multi-pod, multi-container log tailer. Use whenever the task involves looking at logs from a Kubernetes cluster: debugging a failing deployment, checking why pods crash-loop, grepping errors across a namespace, following a rollout, or correlating logs across several pods. Trigger phrases: "check the logs", "tail the logs", "why is this pod failing", "grep the errors in namespace X", "看一下 log", "查 k8s 的 log", "stern", "kubectl logs but for many pods", "logs across all pods of deployment/service", "CrashLoopBackOff logs". Also covers stern's Go templates for reformatting JSON application logs and `--stdin` for replaying a local log file through those templates. NOT for writing application logging code, not for log aggregation backends (Loki/ELK/CloudWatch queries), and not a replacement for `kubectl get`/ `describe` when the question is about pod state rather than log content.4---56# stern78`stern` tails logs from **many pods and many containers at once**, selected by a regex or a9Kubernetes resource, with automatic pickup of new pods and colorized per-pod output.1011## Iron Law1213```14In an agent/non-interactive context, every stern invocation carries --no-follow.15```1617Without it stern **streams forever and never exits** — the tool call blocks until the harness18timeout, and the partial output may be discarded. The only exception: the user explicitly asked to19watch a live stream, and it runs as a background command with a stated stop condition.2021The second half of the law: **bound the volume**. The defaults are `--since 48h` and `--tail -1`22(*every* line ever retained), multiplied by every matching pod. Always narrow at least one of23`--tail` / `--since` — and **both** once the query spans more than a handful of containers, since24they are what truncates each stream. Narrowing *which* containers are read at all (`-E`,25`--exclude-pod`, `-l`, a tighter query) saves more and costs nothing; line filters save neither.2627### Rationalization table2829Every one of these will occur to you mid-task. All of them are wrong.3031| The thought | Reality |32|---|---|33| "This namespace only has two pods, streaming is fine." | Pod count does not make stern exit. Two pods stream forever just as well as fifty. |34| "The user said 'watch'/'monitor'/'看一下現在的狀況', so they want live output." | They want an *answer*. `--no-follow --since 5m` gives it. Streaming into a blocked tool call gives them nothing. |35| "I'll add a timeout, that bounds it." | `timeout` exits 124 and can cut a line mid-write, so the harness may surface a failed command instead of your logs. Bound it with stern's own flags; keep `timeout` only as a belt-and-braces outer limit. |36| "It's idle, it will stop when there's nothing left." | `--no-follow` is the *only* thing that makes stern conclude "all logs shown". Idle means it waits, not exits. |37| "I'll pipe to `head`, so it ends." | It does end, but `head` keeps the *oldest* N lines it happened to see first — the opposite of the recent lines you wanted. `--tail N` selects from the end. |38| "The previous stern call worked without it." | It did not "work" — it hit the timeout, and you saw the partial output that survived. |39| "This is a follow-up to a live session, `--tail 0` is what they meant." | `--tail 0` means "only lines from now on", which by definition never terminates. It is also rejected together with `--no-follow`. |4041The load-bearing test is not "is streaming reasonable here?" but **"who reads the output?"** If the42answer is you, in this tool call, then `--no-follow`.4344## The default command shape4546```bash47stern <query> -n <namespace> --no-follow --tail 50 --since 1h --color never --only-log-lines48```4950| part | why |51|---|---|52| `--no-follow` | terminates; without it the call hangs |53| `--tail 50` | last N lines *per container* — raise deliberately, not by default |54| `--since 1h` | overrides the 48h default |55| `--color never` | no ANSI escapes to confuse parsing/diffing |56| `--only-log-lines` | drops the `+ pod ...` / `- pod ...` attach/detach status lines |5758Keep the status lines (i.e. omit `--only-log-lines`) when you need to know *which pods were even59found* — that is often the actual answer.6061If the task already has a familiar shape — a crash loop, hunting one error across a namespace,62getting output in chronological order, replaying a local log file — take the worked command from63[references/recipes.md](references/recipes.md) instead of composing flags from scratch.6465## Selecting what to read6667The positional argument is a **regex on pod names**, unless it has the form `<resource>/<name>`,68which is an exact resource lookup:6970```bash71stern . --no-follow # every pod in the namespace (regex "." matches all)72stern 'web-\w' --no-follow # web-backend, web-frontend — but not web-12373stern deploy/nginx --no-follow # all pods belonging to that Deployment74stern svc/api --no-follow # all pods behind that Service75```7677Supported resources (with aliases): `pod`/`po`, `replicationcontroller`/`rc`, `service`/`svc`,78`daemonset`/`ds`, `deployment`/`deploy`, `replicaset`/`rs`, `statefulset`/`sts`, `job`.79Plural forms work too (`deployments/nginx`).8081Narrow further with:8283| need | flag |84|---|---|85| one container in multi-container pods | `-c <regex>` |86| drop a sidecar (istio, linkerd…) | `-E istio-proxy` |87| drop noisy pods | `--exclude-pod <regex>` |88| by label | `-l app=api` |89| by field | `--field-selector spec.nodeName=node-1` |90| by node | `--node node-1` |91| several namespaces | `-n a,b` (repeatable) / `-A` for all |92| only containers that have already exited (finished Jobs) | `--container-state terminated` |93| skip init/ephemeral containers | `--init-containers=false`, `--ephemeral-containers=false` — on a mesh-injected pod this also drops the sidecar, see below |9495Since Kubernetes 1.28 a **native sidecar is an init container** with `restartPolicy: Always`, so96`--init-containers=false` on a mesh-injected pod removes the proxy too. That is usually what you97want — but the proxy's access logs carry `x-request-id`, which is what makes following one request98across services possible. Drop `istio-init` alone with `-E istio-init` when you need to keep it.99100That table is the working subset. When you need a flag that is not in it, an exact default, or the101config file that may silently be changing those defaults, read102[references/flags.md](references/flags.md) — guessing a flag name costs a failed invocation.103104`--include` / `-i` and `--exclude` / `-e` filter **log lines** (regex, repeatable) — prefer them over105piping to `grep`, because they apply before the lines are formatted. They do not make the query106cheaper: filtering happens locally on lines already received. To spend less, cut containers with the107selection flags above, or truncate with `--tail`/`--since`.108109```bash110stern deploy/api --no-follow --tail 200 -i 'ERROR|panic' -e 'health.?check'111```112113## Reading the result correctly114115- **Empty output is not an error.** A query matching no pods exits 0 and prints nothing. Verify with116 `kubectl get pods -n <ns>` before concluding "there are no errors".117- **Lines from different pods interleave** in arrival order, not timestamp order. Piping the default118 output to `sort` sorts by *pod name*, because that is what the default template leads with. Either119 serialize the reads, or put the timestamp first with a template:120 ```bash121 stern deploy/api --no-follow --max-log-requests 1 --tail 100 # pod by pod, in order122123 set -o pipefail # sort exits 0 on empty input too124 stern . -A --no-follow --since 10m --only-log-lines --color never -t \125 --template='{{.Message}} @{{.PodName}}/{{.ContainerName}}{{"\n"}}' | sort126 ```127 That works because `-t` prefixes the timestamp **into** `.Message` — which is also why `-t` breaks128 `parseJSON` templates and `-o raw | jq`. There is no separate timestamp field.129- `--max-log-requests` **never silently drops pods.** With `--no-follow` (default 5) it is a130 concurrency limit: every matching container is still read, just fewer at a time, so the output is131 complete and only slower. Without `--no-follow` (default 50) exceeding it is a hard error that132 stops stern with a message naming the flag. The limit counts **containers, not pods** — one133 3-container pod trips `--max-log-requests 1`. Raise it to watch more pods, never because you134 suspect missing output, and **not to go faster**: wall clock is set by `--qps`, not by this flag.135136## Machine-readable output137138```bash139# no pipe: stern's own exit status is the answer140stern deploy/api --no-follow --tail 100 -o json --only-log-lines141142# piped: pipefail belongs in the command, not in a footnote.143# -i '^\s*\{' keeps plain-text framework lines away from jq144set -o pipefail145stern deploy/api --no-follow --tail 100 -o raw --only-log-lines -i '^\s*\{' | jq146```147148Three things in that snippet, each preventing a different wrong answer:149150- **`--only-log-lines`, not a redirect.** The `+ pod › container` attach lines go to stderr, so151 `2>&1 | jq` feeds `jq` a non-JSON line and the pipeline aborts to nothing; `2>/dev/null` avoids152 that but throws away stern's real errors (RBAC `forbidden`, a bad `--context`) along with the153 noise. `--only-log-lines` stops the status lines being printed at all and leaves errors on stderr.154- **`set -o pipefail`.** `jq` exits 0 on empty input, so without it the pipeline reports success even155 when stern failed. It is portable — prefer it. Where you cannot set it, check the *first* command's156 status rather than `$?`: `${PIPESTATUS[0]}` in bash, `${pipestatus[1]}` in zsh (lowercase, and157 1-indexed). Using the bash spelling under zsh expands to an empty string, which reads as "did not158 fail" — the exact failure this bullet exists to prevent, on the shell macOS defaults to.159- **`-i '^\s*\{'` with `-o raw`.** `-o raw` passes the application's own line through, and real160 services mix formats — framework and stdlib lines are plain text alongside the JSON logger. `jq`161 aborts on the first one, prints nothing and exits 5, which `pipefail` turns into a failed pipeline162 for perfectly healthy logs. `-o json` needs no guard: that envelope is always valid JSON. Keep the163 `\s*` — a bare `^\{` drops an indented record silently at exit 0. Guarding on the `jq` side instead164 takes **two** filters, `jq -R 'fromjson? | objects | …'`: `?` guards only `fromjson`, so without165 `objects` a top-level array or scalar parses, reaches the field filter and raises a per-record166 error that `pipefail` cannot see — `jq` reports only the last input's status.167168For the `-o json` field names, the other predefined outputs, and `--template` for reshaping JSON169application logs into something readable, see [references/templates.md](references/templates.md).170171## Preflight1721731. `command -v stern` — if missing, `brew install stern` (macOS/Linux) or174 `kubectl krew install stern`; `kubectl logs` is the fallback for a single pod.1752. Know which cluster you are about to read: `kubectl config current-context`. Confirm with the user176 before pulling logs from an unfamiliar or production-looking context, and remember that log177 content may contain secrets — quote selectively rather than dumping it wholesale.1783. stern honours `$KUBECONFIG`; `--kubeconfig` and `--context` override it.179180## Gotchas181182| symptom | cause / fix |183|---|---|184| command hangs | missing `--no-follow` |185| `Error: --no-follow cannot be used with --tail=0` | `--tail=0` means "only *new* lines", which contradicts exiting; use `--tail 1` or drop `--no-follow` |186| `--condition` rejected | it is only supported with `--tail=0` or `--no-follow` |187| flood of output | `--since 48h` + `--tail -1` defaults across many pods |188| no logs from a crash-looping pod | not a state-filter problem: the default `--container-state all` already covers it, and stern falls back to the last terminated instance's logs. Check `--since`/`--tail` first. `--container-state terminated` would *exclude* it — CrashLoopBackOff is `waiting` |189| a container is skipped entirely | it has no container ID yet (never started — image pull failure, etc.); its logs do not exist, use `kubectl describe pod` |190| a wide query takes ~30s with no error, or logs `"client-side throttling"` | client-go's default rate limiter (QPS 5, burst 10), not stern or the cluster. `--max-log-requests` does **not** affect it — add `--qps=-1`, or `--qps 50 --burst 100` on a shared cluster. At default concurrency the wait is silent, so assume throttling before blaming the API server |191| still slow after `--qps=-1` | transfer volume, not rate limiting. `--include`/`--exclude` filter **client-side**, so a filtered query reads exactly as much as an unfiltered one. Drop whole containers first (`-E`, `--exclude-pod`, `-l`, a tighter query), then truncate what remains with `--tail`/`--since` |192| ANSI garbage in captured output | `--color never` |193| `--timestamps short` gives the long format | the `=` cannot be omitted. `--timestamps short` silently ignores the value and falls back to the full format — no error. Write `--timestamps=short`, or bare `-t` |194| `parseJSON` template suddenly hits its `else` branch | `-t` is on: it prefixes the timestamp into `.Message`. Drop `-t` when parsing JSON |195| empty result from `stern … \| anything` | either `2>&1` merged the stderr status lines into the pipe and the consumer aborted, or stern failed and the consumer returned 0 anyway — `jq`, `sort`, `grep -c`, `wc` all exit 0 on empty input. Use `--only-log-lines` and `set -o pipefail` on every stern pipeline; never `2>/dev/null`, which hides the error that explains it |196| running inside a Pod: forbidden | needs RBAC `get,watch,list` on `pods` and `pods/log` |197198Each reference is linked above from the point where it becomes the right thing to read:199[recipes.md](references/recipes.md) for worked commands, [flags.md](references/flags.md) for the200complete flag set and config file, [templates.md](references/templates.md) for output formatting.