HTTP Log Stats

HTTP Access Log Statistics

Answers measurement questions about HTTP traffic: how many of each status code, how the request durations distribute, how many exceeded a threshold.

This is deliberately separate from ingress-debug, which diagnoses why traffic fails (404 / 502 / TLS / no ADDRESS). Here nothing is broken necessarily — the user wants numbers.

Before you run it: identify the layer

A request usually passes through several components that can each return a status code, and the user's word for the one they mean ("router", "gateway", "entry", "网关") rarely maps to exactly one of them. Counting on the wrong layer produces a confident, wrong answer, so establish the path first:

kubectl get ingress,svc,endpoints -n <ns> | grep <service>

• An Ingress in front → the user's "router 503" is almost certainly the ingress controller's access log. Use this skill.
• No Ingress, traffic reaches the Service directly → there is no access log to count; the numbers must come from the application's own logs or metrics, and the format is application-specific.
• Do not grep the application pod's log for the string 503. Application logs contain the digits 503 in request bodies, upstream error text and unrelated fields; that count is not the number of HTTP 503 responses the entry layer returned.

This step costs one command and routinely saves the entire investigation.

Tool

local_script: cluster="<cluster>", skill="http-log-stats",
              script="http-log-stats.sh", args="<args>"

Runs kubectl from the agent's own context — no debug pod, no node access.

Set cluster to the target cluster's credential name (from cluster_list). local_script resolves it to a kubeconfig and exports KUBECONFIG into the script's environment, so the kubectl calls inside the script pick it up automatically. You may omit it only when the agent is bound to exactly one cluster; with several bound, omitting it is an error listing the available names, not a silent default.

Do not pass --kubeconfig or a KUBECONFIG= prefix in args — cluster selection goes through the tool parameter, and the script takes no such flag. If the name is wrong or the agent is not bound to that cluster, the call fails with the available names; it never silently talks to the wrong cluster.

Parameters

Required:

Optional:

Output

window:    2026-08-17T12:02:06Z .. 2026-08-17T14:02:06Z
source:    live cluster (kubectl logs), ns=ingress-nginx, 3 pod(s)
coverage:  FULL  (earliest retained line 2026-08-17T11:47:33Z)
match:     my-service
parsed:    1007 lines (status 1007, request_time 1007)

status codes:
  200      1000   99.3%
  499         7    0.7%

request_time buckets (seconds):
  <1        983   97.6%
  1-10       17    1.7%
  10-60       7    0.7%
  >=60        0    0.0%

coverage is part of the answer — always report it

A count over a window the log no longer covers is not a smaller truth, it is a wrong number. Never present PARTIAL results as the window total.

Safety

Read-only, and it creates no files — not even a temporary one that is cleaned up afterwards.

• Only two cluster verbs are ever issued: kubectl get pods and kubectl logs. Nothing is created, patched, deleted, or exec'd; no debug pod is spawned.
• No output redirection, no scratch file. Every log line is streamed through a pipe into a single awk that keeps only counters in memory, so a multi-gigabyte window costs a constant amount of RAM and zero disk.
• Shell constructs that create a temp file behind your back are avoided on purpose: no here-doc / here-string (bash backs those with a file in $TMPDIR before 5.1 and beyond the pipe buffer) and no sort (spills to /tmp above its memory threshold). awk formats the report itself.

Notes

• One pull per replica, all dimensions at once. Every bucket and every status code comes out of a single pass. Do not fall back to running kubectl logs | grep -c once per number — on a busy controller each full pull costs 30–90s, so a four-bucket distribution across three replicas becomes twelve full pulls of the same data.
• Absolute window, not a rolling one. --since=2h moves between invocations, so two counts taken minutes apart cover different windows and cannot be compared or summed. That is why --since-time is required.
• All replicas are summed. An ingress controller usually runs several replicas and any of them may have served the request; counting one replica undercounts silently.
• Live cluster is the source. This reads the running controllers, so it is authoritative and current — no collection lag, no sampling, no field mapping in between. When coverage is not FULL, that is the point at which a log or metrics backend becomes the only option; note the switch explicitly when you report, because that data has different freshness and completeness.
• The access log is not necessarily every request. nginx-ingress maps $request_uri to a $loggable flag, so a cluster can switch logging off for chosen paths (health checks are the usual case). What you get is the count of logged requests. If a total looks implausibly low, check log_format / $loggable in the controller's nginx.conf before concluding traffic is missing.
• Buckets, not percentiles. A single streaming pass can count how many requests fell in a range, but it cannot produce an exact p95/p99 without retaining every value. If the user asks for a percentile rather than "how many exceeded N", use the controller's duration histogram (nginx_ingress_controller_request_duration_seconds) — that is what histograms are for.
• Format assumption. Parsing targets the nginx-ingress default log-format-upstream: $status after the quoted request line, $request_time immediately before the bracketed $proxy_upstream_name. If a cluster customises log-format, the script reports that it parsed 0 values rather than returning a silent zero — sample raw lines before trusting anything.

See also

• ingress-debug — why traffic is failing (404 / 502 / TLS / no ADDRESS), as opposed to how much of it there is
• service-debug — backend Service has no healthy endpoints