Splunk Observability K8S Auto Instrumentation Setup

Splunk Observability Kubernetes Auto-Instrumentation

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Wiring zero-code Java, Node.js, Python, .NET, Go, Apache, or Nginx instrumentation into Splunk Observability Cloud APM, adding AlwaysOn Profiling, discovering workloads, or reverting operator-managed instrumentation.
• Preview and review the splunk observability k8s auto instrumentation setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-k8s-auto-instrumentation-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-k8s-auto-instrumentation-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

This skill configures operator-driven, zero-code application auto-instrumentation for Kubernetes workloads that send traces, metrics, and profiling to Splunk Observability Cloud.

It is an overlay on top of splunk-observability-otel-collector-setup. That skill installs the Splunk OTel Collector chart, which in turn installs the OpenTelemetry Operator, the Instrumentation / OpenTelemetryCollector CRDs, and (optionally) a chart-managed default Instrumentation custom resource. This skill takes over from there: it renders one or more fully-configured Instrumentation custom resources per language, writes workload annotation patches that the operator mutating webhook reacts to, and drives the full lifecycle (apply, verify, uninstall).

What it renders

• k8s-instrumentation/instrumentation-cr.yaml — one or more Instrumentation resources (multi-CR when the spec lists multiple).
• k8s-instrumentation/workload-annotations.yaml — strategic-merge patches against spec.template.metadata.annotations (NEVER top-level metadata.annotations) for each target Deployment / StatefulSet / DaemonSet.
• k8s-instrumentation/namespace-annotations.yaml — namespace-level inject-<lang> annotations for namespace-wide opt-in.
• k8s-instrumentation/annotation-backup-configmap.yaml — owned template ConfigMap for versioned, managed-key-only rollback snapshots.
• k8s-instrumentation/annotation-backup.py — reviewed transactional capture/verify/restore planner; it fails before workload mutation on kubectl, JSON, ownership, or completeness errors.
• k8s-instrumentation/obi-daemonset.yaml — only when --enable-obi. Owned ServiceAccount plus digest-pinned Splunk OBI (eBPF) DaemonSet with namespace include/exclude selectors.
• k8s-instrumentation/obi-lifecycle.py — exact OBI ownership/config, rollout, node/kernel coverage, bounded-log validation, and safe purge helper.
• k8s-instrumentation/managed-resource-lifecycle.py — preflights every rendered Instrumentation/OBI/SCC identity before the first mutation, rejects foreign same-name objects, and uses UID/resourceVersion-bound replace/delete operations.
• k8s-instrumentation/openshift-scc-obi.yaml — only when --distribution openshift and --enable-obi. SCC binding for the OBI service account.
• k8s-instrumentation/apply-instrumentation.sh — CRD preflight, SCC then CR then DaemonSet apply, webhook readiness wait.
• k8s-instrumentation/apply-annotations.sh — resolve and merge multi-language targets, transactionally snapshot every managed key, verify the committed backup, then strategic-merge-patch and restart.
• k8s-instrumentation/uninstall.sh — refuse missing/corrupt/incomplete snapshots, restore only rendered managed keys, restart, and optionally purge owned OBI resources and CRs.
• k8s-instrumentation/injection-audit.py — standalone fail-closed audit engine shared by rendered diagnostics.
• k8s-instrumentation/verify-injection.sh — deep check for one or all rendered workloads (exact managed annotations, CR binding, init/sidecar evidence, OTLP env, and language hook).
• k8s-instrumentation/status.sh — one-shot snapshot of CRs, MutatingWebhookConfiguration, operator pod.
• k8s-instrumentation/list-instrumented.sh — runs the same deep drift audit for every rendered target before printing the inventory.
• k8s-instrumentation/preflight-report.md — human-readable preflight verdict.
• discovery/workloads.yaml and discovery/base-collector-probe.json — only when --discover-workloads.
• runbook.md — ordered operator workflow from render through verify and optional uninstall.
• handoff-collector.sh — guidance to run splunk-observability-otel-collector-setup first if CRDs are absent.
• handoff-native-ops.spec.yaml — starter APM detectors spec for splunk-observability-native-ops.
• handoff-dashboard-builder.spec.yaml — APM topology dashboard spec for splunk-observability-dashboard-builder.
• metadata.json — spec digest, preflight verdicts, warning list, rendered file list, target workload list, and namespace-level injection contracts. Workload targets are consumed by --target-all on apply/uninstall; workload and namespace targets are consumed by the deep live auditor.

Safety Rules

• Never ask for a Splunk Observability access token or any other credential in conversation. This skill does not accept credentials on argv or render them. The OpenTelemetry Operator resolves ingest through the $(SPLUNK_OTEL_AGENT) env var injected by the base chart. The optional APM API validation reads a separate mode-0600 User API access-token file through SPLUNK_O11Y_TOKEN_FILE; an ingest-only Org token is not sufficient for that read API.
• Reject direct token flags such as --access-token, --token, --bearer-token, --api-token, --o11y-token, --sf-token, --hec-token, --platform-hec-token, --api-key. Ingest-token handling delegates to the base collector; the read-only APM probe accepts only the dedicated file path from SPLUNK_O11Y_TOKEN_FILE.
• Reject secret-like keys or values in specs, --extra-env, resource attributes, headers, and endpoints (token, HEC, Bearer, Authorization, password, API-key, or Secret material). OTLP URLs must be credential-free HTTP(S) URLs with an explicit port and no userinfo, query, or fragment.
• Image-pull secrets for private registries mirroring ghcr.io/signalfx/* are passed by name (--image-pull-secret <name>); the operator creates the Kubernetes Secret itself and this skill never touches that material.
• Mutating operations are gated: --apply-instrumentation, --apply-annotations, and --uninstall-instrumentation require --accept-auto-instrumentation; CR changes can affect already-annotated workloads on their next restart, while annotation apply/uninstall forces restarts immediately. --apply-instrumentation with --enable-obi also requires --accept-obi-privileged.
• Every live validation or mutation requires --kube-context <name>. Operators who intentionally use kubectl's current context must say so with --allow-current-context; the two flags are mutually exclusive. Non-mutating --dry-run previews do not require either acknowledgement.
• Cluster name is always explicit. This offline renderer never guesses an EKS/GKE/OpenShift identity.
• Operator installation controls (watch namespaces, certificate mode, installation job, or a purported multi-instrumentation feature gate) and vendor webhook detection are not implemented by this overlay. Supplying those legacy inputs fails closed; configure/audit them in the base collector workflow.
• Rendered mutations are repeatable and fail closed on missing preconditions (helm release, CRDs, or an owned complete backup). Annotation apply/uninstall intentionally triggers a new rollout on every invocation and is therefore operationally disruptive even when the annotation values are unchanged.
• Same-name Instrumentation, OBI ServiceAccount/DaemonSet, and OpenShift SCC objects are never adopted implicitly. Apply preflights the full set before its first write; existing objects must carry the exact skill ownership labels, and replacements/deletes are bound to the live UID and resourceVersion. Backup purge requires --target-all, exact snapshot-key coverage, and a UID/resourceVersion-bound ConfigMap delete.

Primary Workflow

1. Confirm the base collector + operator are installed. If not, run splunk-observability-otel-collector-setup first. The rendered handoff-collector.sh carries the exact command.

2. (Optional) Discover candidate workloads. This is read-only; no cluster mutation:

   bash skills/splunk-observability-k8s-auto-instrumentation-setup/scripts/setup.sh \
     --discover-workloads \
     --realm us0 \
     --cluster-name prod-cluster
   

   Then edit splunk-observability-k8s-auto-instrumentation-rendered/discovery/workloads.yaml to mark each workload with its language.

3. Render the overlay assets:

   The CLI-only path is the default; template.example is loaded only when explicitly passed with --spec. This prevents example workloads or policy settings from contaminating an operator's CLI render.

   bash skills/splunk-observability-k8s-auto-instrumentation-setup/scripts/setup.sh \
     --render \
     --realm us0 \
     --cluster-name prod-cluster \
     --deployment-environment prod \
     --languages java,nodejs,python \
     --profiling-enabled \
     --runtime-metrics-enabled \
     --annotate-workload Deployment/prod/payments-api=java \
     --annotate-workload Deployment/prod/checkout-web=nodejs \
     --annotate-workload Deployment/prod/fraud-score=python
   

4. Review splunk-observability-k8s-auto-instrumentation-rendered/:

   • preflight-report.md — every fail / warn / advisory finding.
   • runbook.md — ordered operator steps.
   • k8s-instrumentation/instrumentation-cr.yaml — the CRs that will be applied.
   • k8s-instrumentation/workload-annotations.yaml — the strategic-merge patches.

5. Apply the Instrumentation CR(s) first:

   bash skills/splunk-observability-k8s-auto-instrumentation-setup/scripts/setup.sh \
     --apply-instrumentation \
     --accept-auto-instrumentation \
     --kube-context prod-cluster-admin
   

6. Apply annotations + rollout restart:

   bash skills/splunk-observability-k8s-auto-instrumentation-setup/scripts/setup.sh \
     --apply-annotations \
     --accept-auto-instrumentation \
     --kube-context prod-cluster-admin \
     --target-all
   

7. Verify injection:

   bash skills/splunk-observability-k8s-auto-instrumentation-setup/scripts/validate.sh \
     --check-injection \
     --kube-context prod-cluster-admin
   

Annotation Model

The OpenTelemetry Operator reacts to pod annotations placed at spec.template.metadata.annotations on Deployment / StatefulSet / DaemonSet objects (or at the namespace level). This skill writes strategic-merge patches that target only spec.template.metadata.annotations; it never touches top-level metadata.annotations.

Supported annotations:

• instrumentation.opentelemetry.io/inject-<lang>: "true" — inject using the chart-default Instrumentation CR.
• instrumentation.opentelemetry.io/inject-<lang>: "<namespace>/<crname>" — inject using a specific CR (used for multi-CR / multi-env).
• instrumentation.opentelemetry.io/inject-<lang>: "false" — explicit opt-out for a specific pod.
• instrumentation.opentelemetry.io/container-names: "app,sidecar" — required in Istio-enabled namespaces to avoid instrumenting the istio-proxy sidecar.
• instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: linux-x64 | linux-musl-x64 — Alpine vs glibc for .NET.
• instrumentation.opentelemetry.io/otel-go-auto-target-exe: /path/to/binary — mandatory for Go (eBPF needs the target binary path).

See references/annotation-catalog.md for the full surface, references/annotation-surgery.md for the patching mechanics, and references/instrumentation-cr-reference.md for CR field semantics.

Namespace-level live contract

For each rendered namespace target, live validation requires the Namespace's managed annotations to match metadata exactly. Every non-terminating pod whose phase is not Succeeded or Failed remains in scope and must be Running, Ready, and carry the expected language evidence. A pod-level inject-<language>: "false" is the only exclusion and is itself audited for stale injection artifacts. A pod-level explicit CR value remains in scope and is validated against that rendered CR. Terminal and deleting pods are reported as excluded. A namespace with no active pods fails closed because it provides no production evidence.

namespace-annotations.yaml is declarative input for GitOps or a separately reviewed kubectl apply; the imperative apply-annotations.sh target selector continues to mutate only explicitly rendered workloads.

OBI Behavior

--enable-obi emits an owned ServiceAccount and Splunk OpenTelemetry Zero-code (OBI) DaemonSet with privileged: true, hostPath mounts for /sys/kernel/security and /sys/fs/cgroup, Linux-node selection, and a kernel-≥5.8 requirement. OBI is eBPF-based and instruments compiled binaries (Go, C, Rust) without code or annotation changes. Apply requires --accept-obi-privileged. Because this repository has no audited default for the standalone OBI container contract, render also requires --obi-image ...@sha256:<digest>; tag-only --obi-version input fails closed. On OpenShift, openshift-scc-obi.yaml renders automatically. --check-obi proves the exact digest/config, controller rollout, one Ready OBI pod on every supported Ready schedulable Linux node, and a bounded 10-minute/200-line fatal-rule log window. uninstall.sh --purge-obi verifies exact ownership/config before deleting DaemonSet, optional SCC, then ServiceAccount.

See references/obi-ebpf.md.

Multi-CR / Multi-Environment

When the spec lists more than one Instrumentation CR (e.g. different samplers for dev vs prod), each workload annotation binds to a specific CR via cr=<ns>/<crname>. The OpenTelemetry Operator supports distinct named resources; this overlay does not claim or mutate a separate chart feature gate. Multiple language rows for one workload are intentionally merged into one manifest; duplicate workload/language rows or conflicting shared annotations fail closed.

This skill always renders an explicit <namespace>/<crname> value, including for the default CR, so cross-namespace workloads never depend on the ambiguous bare "true" lookup. --namespace overrides every rendered CR namespace; --instrumentation-cr-name is accepted only for a single-CR render.

Instrumentation image policy

Java, Node.js, Python, .NET, Go, and Apache HTTPD defaults are immutable @sha256 pins copied from the base Collector skill's chart-0.158.0 audited image ledger. Custom image overrides must also be digest-pinned. This repository has no audited Nginx default image, so Nginx fails closed until the operator supplies a reviewed --nginx-image ...@sha256:<digest> override.

Hand-offs

• Base collector + operator + CRDs: splunk-observability-otel-collector-setup. Run first.
• Starter APM detectors: splunk-observability-native-ops. Consume handoff-native-ops.spec.yaml.
• APM topology dashboards: splunk-observability-dashboard-builder. Consume handoff-dashboard-builder.spec.yaml.

Out of scope

• Installing the base collector, operator, CRDs, or cert-manager (handled by the base collector skill).
• Linux / systemd / preload instrumentation (handled by the base collector skill).
• HEC token creation for Splunk Platform logs (handled by splunk-hec-service-setup).
• Application-side SDK wiring (manual instrumentation). This skill is zero-code-only.
• .NET Framework (Windows) auto-instrumentation — not supported by Splunk; explicitly refused.
• Modifying application container images or Dockerfiles; all injection is via the operator init container pattern.
• Splunk On-Call routing and PagerDuty handoff (see splunk-oncall-setup and splunk-observability-native-ops).

Validation

bash skills/splunk-observability-k8s-auto-instrumentation-setup/scripts/validate.sh

Static checks cover:

• YAML well-formedness of every rendered manifest.
• workload-annotations.yaml patches target spec.template.metadata.annotations (not top-level metadata.annotations).
• Go workloads include otel-go-auto-target-exe=.
• No .NET Framework references anywhere.
• Rendered scripts do not echo secrets.
• CR name uniqueness.
• Every Instrumentation language image is pinned by @sha256 and the JSON injection contract exactly matches instrumentation-cr.yaml.
• The rendered backup and OBI helpers byte-match their reviewed source; the OBI metadata contract exactly matches the rendered ServiceAccount/DaemonSet/SCC.

--live --check-apm <service> runs the complete production gate. It enables webhook, Instrumentation CR, injection, backup, optional OBI, and scoped APM checks. A caller may explicitly omit only the APM or backup gate with --skip-apm-check or --skip-backup-check; those named skips are valid only with --live. Individual --check-* flags remain available as narrow diagnostics.

• --check-webhook — exact pinned pod-admission webhook policy/route, CA bundle, Service and ready 9443/TCP EndpointSlice (with a legacy Endpoints fallback only when EndpointSlice is unavailable or empty), Ready Operator pods, and recent webhook error log scan. This proves Kubernetes routing state but does not synthesize an admission request or perform a separate TLS handshake.
• --check-instrumentation — kubectl get otelinst -A shows the rendered CRs with expected fields.
• --check-injection — iterate annotated workloads and namespace contracts. Workload controllers must have observed their current generation and completed rollout; all intended active pods must be Running and Ready with exact language injection evidence, the exact rendered @sha256 init/sidecar image, and expected OTEL_* env. Namespace opt-outs and terminal/deleting exclusions follow the contract above.
• --check-obi — prove the owned OBI manifest contract, digest/config identity, DaemonSet rollout, supported kernel/architecture coverage, Ready pods, and bounded fatal-rule recent logs.
• --check-apm <service> — allow only a service/realm pair bound to a rendered workload in metadata.json, then probe the scoped topology for that service, deployment environment, and Kubernetes cluster.
• --check-backup — every rendered workload has a versioned, owned, structurally complete managed-key snapshot.

Every requested live check is fail-closed: a missing tool, credential, resource, injected pod, backup, API response, or exact APM service is a validation failure. The composed AWS/EKS/O11y staging gate is documented in ../../scripts/staging/README.md.

See reference.md for the full CLI flag reference and the thirteen references/*.md annexes for deep topical documentation.