Kubesense Traces

KubeSense Traces (APM)

Use for anything about requests: latency, p99/p95, error rate, HTTP method, status code, endpoint, service-to-service calls, downstream dependencies. (Log lines and panics → kubesense-logs.)

Requires the KubeSense MCP server; see kubesense-mcp. Tools resolve RBAC against the traces module.

Tools

get-distributed-trace is the highest-leverage tool here. A single span tells you something failed; the trace tells you which hop failed and where the time went.

Field Names — Catalog Labels Only

Storage column names are rejected at every input slot.

Also parse but hidden from discovery: timestamp (→ start_timestamp), duration, primary_workload/primary_namespace (→ perspective_*), associate_workload/associate_namespace (→ partner_*).

service vs workload

Prefer service for application identity — it is the cross-platform identifier (Kubernetes, Docker, legacy). Use workload only when you specifically need Kubernetes topology.

[!NOTE] This is the opposite of the guidance for alert rules, where workload groups more reliably. See kubesense-alerts — the alert engine resolves service against a rollup column that is only populated for SDK-instrumented services.

[!WARNING] node is not valid on traces — it is node_name. (Logs are the reverse: node, not node_name.) Never carry a field name across signals.

There is no body field on traces. Text search over a message is logs-only.

Enum values are lowercase on traces

status = error — lowercase, unlike logs' UPPERCASE type = ERROR.

status_code is a string field, so its values stay quoted strings even when they look numeric: status_code IN (500, 502) yields the strings "500", "502". Range comparisons therefore do not work — enumerate the codes, or use status_code LIKE "5%" for a class.

Read casing from the enum_values column, not example — the example generator returns a blank cell for method, protocol, and status_code.

Duration: ms in, ns out

[!IMPORTANT] Filtering duration takes MILLISECONDS. Aggregating duration returns NANOSECONDS. These are different units in the same field, and getting it wrong is silent.

where: "duration > 500"        →  spans slower than 500 milliseconds

The parser multiplies your value by 1e6 to reach the nanosecond storage column. So:

[!WARNING] The search-traces tool description shows "duration > 250000" as an example. That is wrong — it means 250,000 ms = 250 seconds and will match nothing. Ignore it.

Aggregation is the other direction — p95/avg/max over fields: [{"field": "duration"}] come back in nanoseconds:

Divide by 1e6 to report milliseconds. There is no duration_ms field at the MCP layer — it is rejected as an unknown field.

Trace attributes: group-by only

[!WARNING] @attr in a traces WHERE clause does not work. It is rejected with unknown field "<key>" for signal=traces, because the parser clears the attribute discriminator during the ARRAY_MAP remap and the field then fails column validation. The tool descriptions advertise @db.system = postgresql as working; it does not.

Trace attributes do work everywhere the field is passed structurally — set the flag instead of a prefix:

"group_by_fields": [ { "field": "db.system", "is_attribute": true } ]

Valid in group_by_fields, fields, required_fields, and sort_by. To filter by an attribute value, group by it and read the series labels, or filter on an equivalent top-level column.

(Log attributes are unaffected — @user.id = abc works fine on logs.)

analyze-traces

{
  "from_time": "2026-07-30T10:00:00Z",
  "to_time": "2026-07-30T11:00:00Z",
  "query_type": "range",
  "group_by_fields": [ { "field": "service", "is_attribute": false } ],
  "fields": [ { "field": "duration", "is_attribute": false } ],
  "value_operation": "p95"
}

value_operation: row_count, unique_count, avg, sum, min, max, p50, p90, p95, p99. count is not valid — it is row_count, despite what the schema's summary line suggests.

Error rate as a percentage needs two queries plus a formula → analyze-telemetry. See kubesense-mcp.

Output is columnar JSON: for range, one shared timestamps array plus a positionally aligned values array per series; for instant, one scalar value per series.

search-traces

Same input shape as search-logs. Output is TSV with a # total_count=<N> is_free_search=<bool> header.

Pagination — and its hard limit

Page by narrowing to_time to the minimum start_timestamp returned (not timestamp, which is the span's end time; start_timestamp is the sort and filter key). The window is half-open, so the cursor row is not repeated.

[!WARNING] Exhaustive paging is not achievable for traces. Rows are ordered toStartOfHour(start_timestamp) DESC, observation_point DESC, cluster DESC, start_timestamp DESC. Both observation_point and cluster outrank the timestamp, so pinning clusters alone is not enough — spans at other observation points in the same hour get skipped. And observation_point is not in the trace field catalog, so the where clause cannot pin it (the tool description suggests "observation_point = s-app"; that is rejected as an unknown field).

Practical consequences: keep the window short enough that total_count fits in one page, or use analyze-traces. Treat any paged trace scan as a sample.

Touching duration, trace_id, span_id, or any attribute sets is_free_search and routes to the raw table — slower, no rollups.

get-distributed-trace

The span tree for one request. Start here when investigating a failing or slow request.

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "from_time": "2026-07-30T10:20:00Z",
  "to_time": "2026-07-30T10:40:00Z",
  "max_spans": 200
}

trace_id, from_time, to_time are all required. The window must contain the trace's start_timestamp — traces are time-partitioned, so a window that misses it finds nothing. Use ~10 minutes either side. max_spans defaults to 200, max 500.

[!NOTE] Key on trace_id, not a span's kuid. One trace has many spans; they all share the trace_id.

Reading the output

Block 1 — one row per span in preorder (a span is immediately followed by its children), with a tree column drawing the call chain. Key columns:

Block 2 — ## services, one row per (cluster, workload), ordered worst-first by errors then self time. Read this before the span rows to pick your suspect. It always covers every span, even when block 1 was truncated.

Two diagnostic rules

• A trace with no error may still be the problem. The top row of ## services is the hop that burned the time.
• Large duration_us + small self_time_us = the service was waiting, not working. Investigate what it called (its children), not the service itself. Conversely, if a span errored but every child succeeded, the fault is in that service.

namespace may be blank when spans are attributed to a collector agent rather than the workload's pod; take it from a span row that has one, or from get-workload-detail. Rarely it lists several joined by |, meaning that cluster runs the same workload name in more than one namespace.

Investigation Flow

1. get-distributed-trace(trace_id, ±10min)   → the call chain, errors marked
2. read "## services"                        → who owns the errors / the time
3. search-logs                               → scoped to that workload + the failing
                                               span's start_time/end_time (widen a few s)
4. get-recent-changes / get-infra-issues     → why: deploy, OOM, probe failure
5. analyze-metrics                           → confirm resource pressure

Steps 4–5 use kubesense-infra and kubesense-metrics.

Rules

1. get-trace-or-log-fields with signal: "traces" first, same window as the query.
2. Catalog labels only: service, instance, method, status_code, resource, role, protocol. Never app_service, pod_name, subtype, return_code, clustered_resource, kind, protocol_type.
3. node_name on traces, node on logs. Never carry a field across signals.
4. Prefer service over workload for app identity — but the alert engine prefers workload.
5. duration in WHERE is milliseconds; aggregated duration is nanoseconds. Divide results by 1e6 to report ms. duration_ms does not exist here.
6. @attr filters do not work on traces. Use is_attribute: true in group_by_fields/fields instead.
7. Enum values are lowercase (status = error, role = server); status_code is a string, so enumerate codes or use LIKE "5%" rather than a range.
8. value_operation is row_count, not count.
9. Add role = server when counting requests, or client spans double-count.
10. get-distributed-trace needs a window containing the trace's start; widen it first if nothing is found.
11. Don't try to page traces exhaustively — observation_point can't be pinned. Shorten the window or aggregate.