Splunk Observability OTel Collector Setup
Prerequisites
| Tool or access |
Purpose |
Verify |
| Bash and Python 3 |
Run bundled setup and validation helpers |
bash --version && python3 --version |
| Required product/platform access |
Inspect or configure the selected target |
Complete the documented preflight |
| Credential files for live modes |
Keep secrets out of chat |
Verify paths only |
Workflow Overview
┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘
When to Activate
- Rendering, preflighting, applying, validating, diagnosing, and removing the Splunk Distribution of OpenTelemetry
Collector for Kubernetes and Linux; audit and stage Splunkbase apps 7125, 8698, and 8699 through deployment
servers, Linux.
- Preview and review the splunk observability otel collector setup workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.
Scope
Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.
Examples
Inspect the supported setup modes before selecting one:
bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh --help
Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.
Inspect validation modes before running completion checks:
bash skills/splunk-observability-otel-collector-setup/scripts/validate.sh --help
Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.
Troubleshooting
| Issue |
Cause |
Resolution |
| Preflight fails |
A required tool or access path is missing |
Resolve it before rendering or applying |
| Rendered assets are incomplete |
Required non-secret inputs are absent |
Complete intake and render again |
| Apply is blocked |
Review, credentials, or explicit acceptance is missing |
Use the documented handoff |
| Validation is incomplete |
Live evidence is unavailable |
Record the gap and keep completion open |
Audited baseline
This workflow is pinned and tested against:
- Linux Collector and auto-instrumentation packages
0.158.0.
splunk-otel-collector Helm chart 0.158.0, fetched as the exact GitHub
release archive with SHA-256
088a93ebbcfbecf8e6f7ef3651747b65bbad443f0823489768bd4901cce0a274.
- Chart-selected Collector and auxiliary images are rewritten to the audited
manifest digests recorded in
references/sources.md; unknown custom images
are accepted only when already pinned with @sha256:<64 lowercase hex>.
- Splunkbase apps
7125, 8698, and 8699, version 0.158.0, published
August 7, 2026 and inspected August 20, 2026. App 7125 is the multi-OS/root
artifact; 8698 and 8699 are the Linux and Windows x86_64 split artifacts.
These Splunkbase pins are a separate artifact chain from the Linux collector,
auto-instrumentation, and Helm chart pins above and do not move together.
- Splunk Platform versions explicitly listed for this TA release:
9.0
through 10.5, so --splunk-version 10.5 is accepted. A value outside the
listed trains is still rejected. Omitting --splunk-version skips that
optional compatibility assertion entirely; package audit/rendering on its own
does not certify the package for any specific train.
Treat a newer release as unaudited until --check-upstream, the regression
suite, and the source ledger in references/sources.md have been updated.
Linux rendering always emits executable apply packets, so Collector,
auto-instrumentation, and OBI versions other than these reviewed pins fail
closed rather than producing an unaudited installer.
The workstation or automation host running setup.sh/render_assets.py
requires Python 3.9+. Set PYTHON to an appropriate executable when the system
default is older. Generated Linux target packets have a separate Python 3.6+
minimum and preflight it before token verification or network access. Generated
Kubernetes packets require Helm 3.9+ or Helm 4 and Python 3.8+; Helm 4 uses
the rendered local postrenderer/v1 subprocess plugin.
What this skill owns
The implemented paths are:
- Official Kubernetes Helm deployments, including agent, cluster receiver,
gateway, Windows releases, FIPS image selection, Operator
auto-instrumentation, OBI, Target Allocator, Kubernetes entities/events,
container and Linux-node journald collection, Platform HEC or Splunk Connect
for OTLP log routing, TLS/mTLS file handoff, preflight, rollout checks, and
uninstall.
- Official Linux package-repository installer deployments in agent or gateway
mode, with a pinned installer checksum, local or SSH execution, loopback-safe
agent defaults, stdin token transport, health checks, doctor output, support
bundle, and confirmation-gated uninstall.
- Splunk Add-On for OpenTelemetry Collector (
7125, 8698, 8699) package
audit and staging for deployment servers, Linux heavy forwarders, or Linux
Universal Forwarders, with per-artifact digest pinning, archive hardening,
no-follow local/ preservation, atomic replacement, a private
out-of-app-tree backup, confirmation-gated backup retention, ownership
preservation, and dashboard/no-dashboard evidence. Windows-only 8699
uses deployment-server and Agent Management delivery because local apply
assets are Bash/Python.
- Collector-side custom configuration through a reviewed Linux
--collector-config or guarded Helm --extra-values-file. The skill does not
pretend that an opaque overlay is a fully typed pipeline authoring model.
Native Splunk Observability Metrics Pipeline Management is owned by
splunk-observability-metrics-pipeline-setup; it is downstream aggregation and
routing, not collector pre-ingest processing.
Product routing
Installing a Collector is not the same as proving a product is ready. Route and
validate each requested product explicitly:
| Product or signal |
Base-skill responsibility |
Required completion evidence |
| Infrastructure Monitoring |
Host/Kubernetes metrics, metadata, internal health |
Collector healthy and host/cluster visible in Observability |
| APM |
OTLP trace receiver/export, optional gateway |
Instrumented workload plus a trace visible in APM |
| AlwaysOn Profiling |
Explicit opt-in and supported language agent |
Profile data visible for the intended service |
| Secure Application |
Chart destination only; workload instrumentation still required |
Supported workload annotated/instrumented and security trace visible |
| Kubernetes container/journald/extra-file logs |
Separate typed Platform pipeline plus HEC /services/collector/event or Splunk Connect for OTLP |
Target index receives expected source types and fields; source scope is reviewed |
| Kubernetes events/entities |
Explicit experimental gates |
Event/entity visible; maturity warning recorded |
| Fleet Management / OpAMP |
TA feature-gate handoff only |
Account entitlement and managed Collector visible in Fleet Management |
| Kubernetes zero-code instrumentation |
Delegate to splunk-observability-k8s-auto-instrumentation-setup |
Child-skill workload rollout and trace evidence |
| Database Monitoring |
Delegate to splunk-observability-database-monitoring-setup |
DBMon receiver and UI evidence |
| AI Agent / AI Infrastructure Monitoring |
Delegate to splunk-observability-ai-agent-monitoring-setup |
Instrumented AI workload, evaluation/metric evidence |
| AI Security Monitoring |
Delegate Cisco AI Defense instrumentation to splunk-observability-ai-agent-monitoring-setup |
Security span correlation, licensed integration, and risk UI evidence |
| Browser RUM / Session Replay |
Delegate general setup to splunk-observability-browser-rum-setup; use splunk-observability-k8s-frontend-rum-setup only for Kubernetes frontend injection |
Browser beacon and UI evidence |
| Mobile RUM |
Delegate to splunk-observability-mobile-rum-setup; mobile beacons bypass this Collector |
Mobile session/beacon and UI evidence |
| Synthetics |
Delegate to splunk-observability-synthetics-setup |
Test run, result, and detector evidence |
| SLOs |
Delegate to splunk-observability-slo-setup |
SLI data, SLO calculation, and alert evidence |
| Dashboards and detectors |
Delegate to splunk-observability-dashboard-builder and splunk-observability-native-ops |
Dashboard population and detector state |
| DXA, AI Assistant, Observability Mobile, Related Content, and deep product UI |
Delegate to splunk-observability-deep-native-workflows |
Product-specific navigation and populated UI evidence |
| SignalFlow and data tools |
Delegate native operations to splunk-observability-native-ops / splunk-observability-deep-native-workflows |
Executed analytics or metadata workflow evidence |
| ITSI / ITE Work / App for Content Packs |
Separate Splunk Platform workflows: splunk-itsi-setup and splunk-itsi-config |
Platform app/content-pack and ITSI object evidence |
| Metrics Pipeline Management |
Delegate to splunk-observability-metrics-pipeline-setup |
Rule-set and post-rule metric evidence |
| AWS Lambda APM |
Delegate to splunk-observability-aws-lambda-apm-setup |
Instrumented invocation and trace evidence |
| Coding agents |
Delegate to splunk-observability-coding-agent-instrumentation-setup |
Agent telemetry at every requested destination |
| ThousandEyes |
Delegate to splunk-observability-thousandeyes-integration |
Linked test/metric and dashboard evidence |
| Splunk Connect for OTLP |
Delegate receiver-side setup to splunk-connect-for-otlp-setup |
OTLP receiver health and target-index evidence |
| Network Explorer |
--enable-network-explorer enforces the supported one-replica gateway profile and renders the separate upstream eBPF-chart handoff |
eBPF DaemonSet, representative tcp.*/udp.*/dns.*/http.* metrics, and populated Network Explorer UI evidence |
See references/coverage.md for deployment-method and feature classification.
Non-negotiable safety rules
- Never request or render a token value. Accept only paths to token files.
- Reject direct and
--flag=value token arguments without echoing their value.
- Token and private-key files must be single-link, non-symlink regular files,
nonempty, mode
600, and contain no NUL, newline, or whitespace. Tokens are
capped at 16 KiB and use only the environment/config-safe
A-Za-z0-9._~+/=- alphabet, which includes the documented base64 token
characters. Linux reads one no-follow descriptor into memory and never
rereads the source path or creates a temporary token file.
- Rendered base values and copied overlays are integrity-bound. The only mutable
values overlay is a schema-constrained Secret revision annotation.
- Generated files, including root
metadata.json, are published through
same-directory atomic replacement and refuse an existing final-component
symlink. A render never follows that symlink into an arbitrary target.
- Existing Kubernetes Secrets and PriorityClasses are mutated or deleted only
when exact skill/release/namespace ownership annotations match. Create uses
an atomic create, updates use UID/resourceVersion-bound replace, and deletes
use UID/resourceVersion preconditions so a concurrent replacement cannot be
adopted or removed. Secret and PriorityClass cleanup each require a separate
confirmation variable.
- Keep
secret.create=false; create the Kubernetes Secret from files. Guarded
extra values may not override secret creation or contain inline secret keys.
- Pin the chart, Collector, Linux installer URL, and Linux installer SHA-256.
A Linux installer mirror may change the HTTPS URL, but executable packets
still require the exact audited digest; no arbitrary digest override exists.
- Download the chart archive once into the packet cache, verify it before every
use, and pass that same local archive to preflight and install. The
integrity-bound post-renderer must replace every audited mutable image with
its digest, reject unknown tags/digests in audited repositories, and reject
any unpinned custom image. Status rechecks live workload, Instrumentation,
and pod specs so an admission-time image rewrite fails validation.
- Before Helm preflight, install, status, or uninstall, inspect the exact-name
release across all common Helm 3/4 statuses and require the rendered
namespace plus a
splunk-otel-collector-* chart identity. Installation
accepts only an absent or deployed owned release; uninstall accepts only an
owned deployed/failed release. Never replace or delete a foreign same-name
chart.
- When the chart installation Job owns the Instrumentation CR, refuse foreign
ownership, snapshot an existing owned CR and Helm revision before mutation,
and require the post-install revision to be exactly the expected successor.
If ownership validation fails, roll an existing release back to the captured
revision or uninstall a new release, then restore the CR. A concurrent Helm
revision refuses automatic rollback and retains the recovery snapshot.
Repeat ownership checks before status or post-Helm-uninstall cleanup.
- The audited installation Job uses kubectl
v1.35.1, so its executable
compatibility gate covers Kubernetes server minors 1.34 through 1.36.
Outside that range, audit and pin a matching image in a future skill update.
Disabling the Job selects upstream resource mode; Helm 4 first install is
rejected and must use the upstream two-step operator/webhook-ready handoff.
- The Linux agent binds to
127.0.0.1 unless the operator explicitly chooses a
broader interface. Gateway mode uses the upstream 0.0.0.0 default.
- Do not generate removed
--trace-url or deprecated --hec-url installer
flags. The tagged installer exposes native-host Platform flags, but its token
option is argv-based; this workflow refuses to put a HEC token on argv and
routes native-host Platform data through a reviewed custom config or UF/TA
handoff.
- SSH install streams the token over stdin and never copies it to a remote file.
- Linux local/SSH preflight requires Bash, curl, Python 3.6+, tar, a SHA-256
tool, active systemd, the pinned installer's system-account utilities, the
matching
apt-get, yum/dnf, or zypper package tools, and root or
passwordless noninteractive sudo. It enforces the tagged installer's exact
distro/version and amd64/arm64 matrix. OBI additionally requires
sha256sum and gzip to already exist so preflight never installs a package.
It rejects an existing install and proves custom-config
traversal/readability before package mutation.
- Linux OBI install and status re-hash the extracted
v0.6.0 executable
against the independently audited amd64 or arm64 binary digest after the
upstream release-archive checksum succeeds.
- Linux status, doctor, and support-bundle helpers hash-verify the generated
redactor and fail closed when privileged collection or redaction fails.
Doctor results distinguish complete/healthy (
0), complete/unhealthy (1),
and diagnostics-incomplete (2) with a matching final completion marker.
Support bundles publish complete evidence for both healthy and unhealthy
results, but refuse incomplete or marker-mismatched diagnostics. Bundles
record diagnostic-state.txt, use a private staging directory, mode 600,
atomic no-replace publication, and reject existing or symlink output paths.
- Linux uninstall requires a second explicit confirmation before allowing the
upstream uninstaller to remove detected auto-instrumentation, deletes only the
installer-generated token-bearing environment files, and still requires token
revocation as a separate operating step.
--apply-ta must not use placeholder secret mode. Placeholder templates are
disabled and render-only.
- TA preflight, staging, local-overlay apply, and backup management require an
external Python
3.6 or newer interpreter with os.O_NOFOLLOW and
os.O_DIRECTORY; preflight proves this before package work begins.
- Splunkbase marks the audited TA artifacts FIPS-incompatible. FedRAMP status
is not a Splunkbase metadata field; report it as not documented, not as a
validated false claim.
--fips-enabled selects and verifies the audited Kubernetes FIPS image; it
does not certify a FedRAMP deployment. Splunk's previously indexed FedRAMP
draft included hosting, agent, instrumentation, and integration limits, but
its dedicated public Help URL currently returns 404 and contained
contradictory authorization wording. Verify the live FedRAMP Marketplace
package, order/contract, hosting boundary, and supported-feature list with
Splunk and the compliance owner instead of treating this packet as evidence.
- Do not enable Splunk Platform traces without
--accept-experimental-platform-traces; chart capability and product support
documentation currently conflict.
Workflow
- Determine the target and requested products/signals. Do not use
--all-signals as a substitute for product discovery.
- Collect only non-secret inputs: realm, topology, cluster/host identity,
destination endpoints/indexes, and secret-file paths.
- Render first.
- Review
metadata.json, warnings, values/config, package audit, and exact
apply commands.
- Run static validation. Use
--check-upstream when network access is
available.
- Apply only after explicit authorization.
- Run the rendered status/doctor workflow and prove backend telemetry for each
enabled product. Record
configured, instrumented, telemetry observed,
and product UI verified separately.
Safe defaults
- Kubernetes: metrics and traces on; container logs, journald, profiling, events, discovery,
Operator auto-instrumentation, OBI, Secure Application, entities, and Target
Allocator off. Agent and cluster receiver on. Gateway off; when enabled, three
replicas.
- Linux: agent mode,
512 MiB, loopback bind, no discovery, no
auto-instrumentation, no profiling, no SDK metric/log exporter overrides, and
no OBI.
- TA: render only, deployment-server target, agent mode, placeholder secret
mode, official digest required for actionable output, and a no-match
server-class whitelist until the operator supplies a reviewed client filter.
Render examples
Kubernetes:
bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \
--render-k8s \
--realm us0 \
--cluster-name production-cluster \
--chart-version 0.158.0 \
--o11y-token-file /secure/splunk_o11y_token
Kubernetes logs through HEC:
bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \
--render-k8s \
--realm us0 \
--cluster-name production-cluster \
--enable-logs \
--platform-hec-url https://splunk.example.com:8088/services/collector/event \
--platform-hec-index k8s_logs \
--platform-hec-token-file /secure/splunk_hec_token \
--o11y-token-file /secure/splunk_o11y_token
Platform-only Kubernetes logs through Splunk Connect for OTLP:
bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \
--render-k8s \
--cluster-name production-cluster \
--disable-metrics --disable-traces \
--enable-logs \
--platform-otlp-endpoint splunk-otlp.example.com:4317
Linux:
bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \
--render-linux \
--realm us0 \
--o11y-token-file /secure/splunk_o11y_token
TA package audit (use the matching 7125/8698/8699 filename):
bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \
--render-ta \
--realm us0 \
--ta-package-path ./splunk-add-on-for-opentelemetry-collector_01542.tgz \
--ta-target deployment-server \
--ta-serverclass-whitelist 'otel-uf-*'
template.example is a manual intake worksheet, not an executable spec. Map
reviewed values to CLI flags; never assume the setup script reads
template.local.
Validation
bash skills/splunk-observability-otel-collector-setup/scripts/validate.sh \
--check-k8s --check-linux --output-dir splunk-observability-otel-rendered
bash skills/splunk-observability-otel-collector-setup/scripts/validate.sh \
--check-k8s --check-linux --check-upstream \
--output-dir splunk-observability-otel-rendered
# Live controller, pod-readiness, and audited-image checks without reading
# Helm release Secrets:
bash skills/splunk-observability-otel-collector-setup/scripts/validate.sh \
--output-dir splunk-observability-otel-rendered \
--k8s-workloads-only --kube-context CONTEXT
Full live status additionally enforces supported kubectl/API-server version
skew and scans both primary Collector pods and auxiliary chart pods. Confirmed
SignalFx conversion drops caused by the 36-dimension limit fail the gate as
telemetry loss. Log-retrieval failures and matched log bodies are suppressed;
only rule counts are reported. Both live paths require rendered core container
names/controllers to retain their exact audited image pins while allowing
unrelated auxiliary containers only when their images are digest-pinned. The
workload-only path unions both label inventories by Pod identity, then fetches
each Pod once for a coherent readiness/image snapshot. It intentionally does
not read logs and therefore cannot provide log-loss evidence.
The repository-wide AWS/EKS/O11y staging gate composes this secret-free mode
with AWS identity, EKS endpoint, auto-instrumentation, APM, and AWS integration
checks. See ../../scripts/staging/README.md.
TA Completion Gate
For TA/add-on work, also follow
../shared/ta_completion_gate.md. The
data ingest path must be configured and validated. Discover any
pre-built/package-shipped dashboards, then prove they are visible,
macro-aligned, and returning data. If the package ships no dashboards, record
that explicit package evidence. The currently audited TA source/package family
records no shipped data/ui/views; completion therefore depends on _internal
diagnostics and Observability/Platform telemetry rather than a nonexistent
packaged dashboard.
Read reference.md for the option contract, lifecycle behavior, known product
documentation conflicts, and explicit handoffs.
1---2name: splunk-observability-otel-collector-setup3description: "Use when rendering, preflighting, applying, validating, diagnosing, and removing the Splunk Distribution4 of OpenTelemetry Collector for Kubernetes and Linux; audit and stage Splunkbase apps 7125, 8698, and5 8699 through deployment servers, Linux heavy forwarders, or Linux Universal Forwarders; configure6 guarded Splunk Platform HEC or Splunk Connect for OTLP destinations; and route specialized Observability7 products to their owning skills."8compatibility: "Splunk Cloud Platform 10.5.2605: conditional. Follow documented package, entitlement, topology, and customer-managed runtime guardrails; self-managed paths remain on the public 10.4 baseline."9metadata:10 source: https://skillmd.com/skills/chambear2809/splunk-observability-otel-collector-setup11 splunk_cloud_10_5: "conditional"12 compatibility_verified: "2026-08-20"13---1415# Splunk Observability OTel Collector Setup1617## Prerequisites1819| Tool or access | Purpose | Verify |20|---|---|---|21| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |22| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |23| Credential files for live modes | Keep secrets out of chat | Verify paths only |2425## Workflow Overview2627```text28┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐29│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │30└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘31```3233## When to Activate3435- Rendering, preflighting, applying, validating, diagnosing, and removing the Splunk Distribution of OpenTelemetry36 Collector for Kubernetes and Linux; audit and stage Splunkbase apps 7125, 8698, and 8699 through deployment37 servers, Linux.38- Preview and review the splunk observability otel collector setup workflow before any live apply phase.39- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.4041## Scope4243Follow the documented read-only or render-first path whenever it is available.44This skill does not imply permission to mutate live systems. Require explicit45apply flags, protected credentials, and operator review for state changes.4647## Examples4849Inspect the supported setup modes before selecting one:5051```bash52bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh --help53```5455Expected output: usage, supported modes, and required arguments are displayed56without changing the target environment.5758Inspect validation modes before running completion checks:5960```bash61bash skills/splunk-observability-otel-collector-setup/scripts/validate.sh --help62```6364Expected output: offline, live, and completion options are displayed when the65skill supports them; help exits without mutation.6667## Troubleshooting6869| Issue | Cause | Resolution |70|---|---|---|71| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |72| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |73| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |74| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |7576## Audited baseline7778This workflow is pinned and tested against:7980- Linux Collector and auto-instrumentation packages `0.158.0`.81- `splunk-otel-collector` Helm chart `0.158.0`, fetched as the exact GitHub82 release archive with SHA-25683 `088a93ebbcfbecf8e6f7ef3651747b65bbad443f0823489768bd4901cce0a274`.84- Chart-selected Collector and auxiliary images are rewritten to the audited85 manifest digests recorded in `references/sources.md`; unknown custom images86 are accepted only when already pinned with `@sha256:<64 lowercase hex>`.87- Splunkbase apps `7125`, `8698`, and `8699`, version `0.158.0`, published88 August 7, 2026 and inspected August 20, 2026. App `7125` is the multi-OS/root89 artifact; `8698` and `8699` are the Linux and Windows x86_64 split artifacts.90 These Splunkbase pins are a separate artifact chain from the Linux collector,91 auto-instrumentation, and Helm chart pins above and do not move together.92- Splunk Platform versions explicitly listed for this TA release: `9.0`93 through `10.5`, so `--splunk-version 10.5` is accepted. A value outside the94 listed trains is still rejected. Omitting `--splunk-version` skips that95 optional compatibility assertion entirely; package audit/rendering on its own96 does not certify the package for any specific train.9798Treat a newer release as unaudited until `--check-upstream`, the regression99suite, and the source ledger in `references/sources.md` have been updated.100Linux rendering always emits executable apply packets, so Collector,101auto-instrumentation, and OBI versions other than these reviewed pins fail102closed rather than producing an unaudited installer.103104The workstation or automation host running `setup.sh`/`render_assets.py`105requires Python 3.9+. Set `PYTHON` to an appropriate executable when the system106default is older. Generated Linux target packets have a separate Python 3.6+107minimum and preflight it before token verification or network access. Generated108Kubernetes packets require Helm `3.9+` or Helm `4` and Python `3.8+`; Helm 4 uses109the rendered local `postrenderer/v1` subprocess plugin.110111## What this skill owns112113The implemented paths are:1141151. Official Kubernetes Helm deployments, including agent, cluster receiver,116 gateway, Windows releases, FIPS image selection, Operator117 auto-instrumentation, OBI, Target Allocator, Kubernetes entities/events,118 container and Linux-node journald collection, Platform HEC or Splunk Connect119 for OTLP log routing, TLS/mTLS file handoff, preflight, rollout checks, and120 uninstall.1212. Official Linux package-repository installer deployments in agent or gateway122 mode, with a pinned installer checksum, local or SSH execution, loopback-safe123 agent defaults, stdin token transport, health checks, doctor output, support124 bundle, and confirmation-gated uninstall.1253. Splunk Add-On for OpenTelemetry Collector (`7125`, `8698`, `8699`) package126 audit and staging for deployment servers, Linux heavy forwarders, or Linux127 Universal Forwarders, with per-artifact digest pinning, archive hardening,128 no-follow `local/` preservation, atomic replacement, a private129 out-of-app-tree backup, confirmation-gated backup retention, ownership130 preservation, and dashboard/no-dashboard evidence. Windows-only `8699`131 uses deployment-server and Agent Management delivery because local apply132 assets are Bash/Python.1334. Collector-side custom configuration through a reviewed Linux134 `--collector-config` or guarded Helm `--extra-values-file`. The skill does not135 pretend that an opaque overlay is a fully typed pipeline authoring model.136137Native Splunk Observability Metrics Pipeline Management is owned by138`splunk-observability-metrics-pipeline-setup`; it is downstream aggregation and139routing, not collector pre-ingest processing.140141## Product routing142143Installing a Collector is not the same as proving a product is ready. Route and144validate each requested product explicitly:145146| Product or signal | Base-skill responsibility | Required completion evidence |147|---|---|---|148| Infrastructure Monitoring | Host/Kubernetes metrics, metadata, internal health | Collector healthy and host/cluster visible in Observability |149| APM | OTLP trace receiver/export, optional gateway | Instrumented workload plus a trace visible in APM |150| AlwaysOn Profiling | Explicit opt-in and supported language agent | Profile data visible for the intended service |151| Secure Application | Chart destination only; workload instrumentation still required | Supported workload annotated/instrumented and security trace visible |152| Kubernetes container/journald/extra-file logs | Separate typed Platform pipeline plus HEC `/services/collector/event` or Splunk Connect for OTLP | Target index receives expected source types and fields; source scope is reviewed |153| Kubernetes events/entities | Explicit experimental gates | Event/entity visible; maturity warning recorded |154| Fleet Management / OpAMP | TA feature-gate handoff only | Account entitlement and managed Collector visible in Fleet Management |155| Kubernetes zero-code instrumentation | Delegate to `splunk-observability-k8s-auto-instrumentation-setup` | Child-skill workload rollout and trace evidence |156| Database Monitoring | Delegate to `splunk-observability-database-monitoring-setup` | DBMon receiver and UI evidence |157| AI Agent / AI Infrastructure Monitoring | Delegate to `splunk-observability-ai-agent-monitoring-setup` | Instrumented AI workload, evaluation/metric evidence |158| AI Security Monitoring | Delegate Cisco AI Defense instrumentation to `splunk-observability-ai-agent-monitoring-setup` | Security span correlation, licensed integration, and risk UI evidence |159| Browser RUM / Session Replay | Delegate general setup to `splunk-observability-browser-rum-setup`; use `splunk-observability-k8s-frontend-rum-setup` only for Kubernetes frontend injection | Browser beacon and UI evidence |160| Mobile RUM | Delegate to `splunk-observability-mobile-rum-setup`; mobile beacons bypass this Collector | Mobile session/beacon and UI evidence |161| Synthetics | Delegate to `splunk-observability-synthetics-setup` | Test run, result, and detector evidence |162| SLOs | Delegate to `splunk-observability-slo-setup` | SLI data, SLO calculation, and alert evidence |163| Dashboards and detectors | Delegate to `splunk-observability-dashboard-builder` and `splunk-observability-native-ops` | Dashboard population and detector state |164| DXA, AI Assistant, Observability Mobile, Related Content, and deep product UI | Delegate to `splunk-observability-deep-native-workflows` | Product-specific navigation and populated UI evidence |165| SignalFlow and data tools | Delegate native operations to `splunk-observability-native-ops` / `splunk-observability-deep-native-workflows` | Executed analytics or metadata workflow evidence |166| ITSI / ITE Work / App for Content Packs | Separate Splunk Platform workflows: `splunk-itsi-setup` and `splunk-itsi-config` | Platform app/content-pack and ITSI object evidence |167| Metrics Pipeline Management | Delegate to `splunk-observability-metrics-pipeline-setup` | Rule-set and post-rule metric evidence |168| AWS Lambda APM | Delegate to `splunk-observability-aws-lambda-apm-setup` | Instrumented invocation and trace evidence |169| Coding agents | Delegate to `splunk-observability-coding-agent-instrumentation-setup` | Agent telemetry at every requested destination |170| ThousandEyes | Delegate to `splunk-observability-thousandeyes-integration` | Linked test/metric and dashboard evidence |171| Splunk Connect for OTLP | Delegate receiver-side setup to `splunk-connect-for-otlp-setup` | OTLP receiver health and target-index evidence |172| Network Explorer | `--enable-network-explorer` enforces the supported one-replica gateway profile and renders the separate upstream eBPF-chart handoff | eBPF DaemonSet, representative `tcp.*`/`udp.*`/`dns.*`/`http.*` metrics, and populated Network Explorer UI evidence |173174See `references/coverage.md` for deployment-method and feature classification.175176## Non-negotiable safety rules177178- Never request or render a token value. Accept only paths to token files.179- Reject direct and `--flag=value` token arguments without echoing their value.180- Token and private-key files must be single-link, non-symlink regular files,181 nonempty, mode `600`, and contain no NUL, newline, or whitespace. Tokens are182 capped at 16 KiB and use only the environment/config-safe183 `A-Za-z0-9._~+/=-` alphabet, which includes the documented base64 token184 characters. Linux reads one no-follow descriptor into memory and never185 rereads the source path or creates a temporary token file.186- Rendered base values and copied overlays are integrity-bound. The only mutable187 values overlay is a schema-constrained Secret revision annotation.188- Generated files, including root `metadata.json`, are published through189 same-directory atomic replacement and refuse an existing final-component190 symlink. A render never follows that symlink into an arbitrary target.191- Existing Kubernetes Secrets and PriorityClasses are mutated or deleted only192 when exact skill/release/namespace ownership annotations match. Create uses193 an atomic create, updates use UID/resourceVersion-bound replace, and deletes194 use UID/resourceVersion preconditions so a concurrent replacement cannot be195 adopted or removed. Secret and PriorityClass cleanup each require a separate196 confirmation variable.197- Keep `secret.create=false`; create the Kubernetes Secret from files. Guarded198 extra values may not override secret creation or contain inline secret keys.199- Pin the chart, Collector, Linux installer URL, and Linux installer SHA-256.200 A Linux installer mirror may change the HTTPS URL, but executable packets201 still require the exact audited digest; no arbitrary digest override exists.202- Download the chart archive once into the packet cache, verify it before every203 use, and pass that same local archive to preflight and install. The204 integrity-bound post-renderer must replace every audited mutable image with205 its digest, reject unknown tags/digests in audited repositories, and reject206 any unpinned custom image. Status rechecks live workload, Instrumentation,207 and pod specs so an admission-time image rewrite fails validation.208- Before Helm preflight, install, status, or uninstall, inspect the exact-name209 release across all common Helm 3/4 statuses and require the rendered210 namespace plus a `splunk-otel-collector-*` chart identity. Installation211 accepts only an absent or deployed owned release; uninstall accepts only an212 owned deployed/failed release. Never replace or delete a foreign same-name213 chart.214- When the chart installation Job owns the Instrumentation CR, refuse foreign215 ownership, snapshot an existing owned CR and Helm revision before mutation,216 and require the post-install revision to be exactly the expected successor.217 If ownership validation fails, roll an existing release back to the captured218 revision or uninstall a new release, then restore the CR. A concurrent Helm219 revision refuses automatic rollback and retains the recovery snapshot.220 Repeat ownership checks before status or post-Helm-uninstall cleanup.221- The audited installation Job uses kubectl `v1.35.1`, so its executable222 compatibility gate covers Kubernetes server minors `1.34` through `1.36`.223 Outside that range, audit and pin a matching image in a future skill update.224 Disabling the Job selects upstream resource mode; Helm 4 first install is225 rejected and must use the upstream two-step operator/webhook-ready handoff.226- The Linux agent binds to `127.0.0.1` unless the operator explicitly chooses a227 broader interface. Gateway mode uses the upstream `0.0.0.0` default.228- Do not generate removed `--trace-url` or deprecated `--hec-url` installer229 flags. The tagged installer exposes native-host Platform flags, but its token230 option is argv-based; this workflow refuses to put a HEC token on argv and231 routes native-host Platform data through a reviewed custom config or UF/TA232 handoff.233- SSH install streams the token over stdin and never copies it to a remote file.234- Linux local/SSH preflight requires Bash, curl, Python 3.6+, tar, a SHA-256235 tool, active systemd, the pinned installer's system-account utilities, the236 matching `apt-get`, `yum`/`dnf`, or `zypper` package tools, and root or237 passwordless noninteractive sudo. It enforces the tagged installer's exact238 distro/version and `amd64`/`arm64` matrix. OBI additionally requires239 `sha256sum` and `gzip` to already exist so preflight never installs a package.240 It rejects an existing install and proves custom-config241 traversal/readability before package mutation.242- Linux OBI install and status re-hash the extracted `v0.6.0` executable243 against the independently audited `amd64` or `arm64` binary digest after the244 upstream release-archive checksum succeeds.245- Linux status, doctor, and support-bundle helpers hash-verify the generated246 redactor and fail closed when privileged collection or redaction fails.247 Doctor results distinguish complete/healthy (`0`), complete/unhealthy (`1`),248 and diagnostics-incomplete (`2`) with a matching final completion marker.249 Support bundles publish complete evidence for both healthy and unhealthy250 results, but refuse incomplete or marker-mismatched diagnostics. Bundles251 record `diagnostic-state.txt`, use a private staging directory, mode `600`,252 atomic no-replace publication, and reject existing or symlink output paths.253- Linux uninstall requires a second explicit confirmation before allowing the254 upstream uninstaller to remove detected auto-instrumentation, deletes only the255 installer-generated token-bearing environment files, and still requires token256 revocation as a separate operating step.257- `--apply-ta` must not use `placeholder` secret mode. Placeholder templates are258 disabled and render-only.259- TA preflight, staging, local-overlay apply, and backup management require an260 external Python `3.6` or newer interpreter with `os.O_NOFOLLOW` and261 `os.O_DIRECTORY`; preflight proves this before package work begins.262- Splunkbase marks the audited TA artifacts FIPS-incompatible. FedRAMP status263 is not a Splunkbase metadata field; report it as not documented, not as a264 validated false claim.265- `--fips-enabled` selects and verifies the audited Kubernetes FIPS image; it266 does not certify a FedRAMP deployment. Splunk's previously indexed FedRAMP267 draft included hosting, agent, instrumentation, and integration limits, but268 its dedicated public Help URL currently returns `404` and contained269 contradictory authorization wording. Verify the live FedRAMP Marketplace270 package, order/contract, hosting boundary, and supported-feature list with271 Splunk and the compliance owner instead of treating this packet as evidence.272- Do not enable Splunk Platform traces without273 `--accept-experimental-platform-traces`; chart capability and product support274 documentation currently conflict.275276## Workflow2772781. Determine the target and requested products/signals. Do not use279 `--all-signals` as a substitute for product discovery.2802. Collect only non-secret inputs: realm, topology, cluster/host identity,281 destination endpoints/indexes, and secret-file paths.2823. Render first.2834. Review `metadata.json`, warnings, values/config, package audit, and exact284 apply commands.2855. Run static validation. Use `--check-upstream` when network access is286 available.2876. Apply only after explicit authorization.2887. Run the rendered status/doctor workflow and prove backend telemetry for each289 enabled product. Record `configured`, `instrumented`, `telemetry observed`,290 and `product UI verified` separately.291292## Safe defaults293294- Kubernetes: metrics and traces on; container logs, journald, profiling, events, discovery,295 Operator auto-instrumentation, OBI, Secure Application, entities, and Target296 Allocator off. Agent and cluster receiver on. Gateway off; when enabled, three297 replicas.298- Linux: agent mode, `512` MiB, loopback bind, no discovery, no299 auto-instrumentation, no profiling, no SDK metric/log exporter overrides, and300 no OBI.301- TA: render only, deployment-server target, agent mode, placeholder secret302 mode, official digest required for actionable output, and a no-match303 server-class whitelist until the operator supplies a reviewed client filter.304305## Render examples306307Kubernetes:308309```bash310bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \311 --render-k8s \312 --realm us0 \313 --cluster-name production-cluster \314 --chart-version 0.158.0 \315 --o11y-token-file /secure/splunk_o11y_token316```317318Kubernetes logs through HEC:319320```bash321bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \322 --render-k8s \323 --realm us0 \324 --cluster-name production-cluster \325 --enable-logs \326 --platform-hec-url https://splunk.example.com:8088/services/collector/event \327 --platform-hec-index k8s_logs \328 --platform-hec-token-file /secure/splunk_hec_token \329 --o11y-token-file /secure/splunk_o11y_token330```331332Platform-only Kubernetes logs through Splunk Connect for OTLP:333334```bash335bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \336 --render-k8s \337 --cluster-name production-cluster \338 --disable-metrics --disable-traces \339 --enable-logs \340 --platform-otlp-endpoint splunk-otlp.example.com:4317341```342343Linux:344345```bash346bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \347 --render-linux \348 --realm us0 \349 --o11y-token-file /secure/splunk_o11y_token350```351352TA package audit (use the matching 7125/8698/8699 filename):353354```bash355bash skills/splunk-observability-otel-collector-setup/scripts/setup.sh \356 --render-ta \357 --realm us0 \358 --ta-package-path ./splunk-add-on-for-opentelemetry-collector_01542.tgz \359 --ta-target deployment-server \360 --ta-serverclass-whitelist 'otel-uf-*'361```362363`template.example` is a manual intake worksheet, not an executable spec. Map364reviewed values to CLI flags; never assume the setup script reads365`template.local`.366367## Validation368369```bash370bash skills/splunk-observability-otel-collector-setup/scripts/validate.sh \371 --check-k8s --check-linux --output-dir splunk-observability-otel-rendered372373bash skills/splunk-observability-otel-collector-setup/scripts/validate.sh \374 --check-k8s --check-linux --check-upstream \375 --output-dir splunk-observability-otel-rendered376377# Live controller, pod-readiness, and audited-image checks without reading378# Helm release Secrets:379bash skills/splunk-observability-otel-collector-setup/scripts/validate.sh \380 --output-dir splunk-observability-otel-rendered \381 --k8s-workloads-only --kube-context CONTEXT382```383384Full live status additionally enforces supported kubectl/API-server version385skew and scans both primary Collector pods and auxiliary chart pods. Confirmed386SignalFx conversion drops caused by the 36-dimension limit fail the gate as387telemetry loss. Log-retrieval failures and matched log bodies are suppressed;388only rule counts are reported. Both live paths require rendered core container389names/controllers to retain their exact audited image pins while allowing390unrelated auxiliary containers only when their images are digest-pinned. The391workload-only path unions both label inventories by Pod identity, then fetches392each Pod once for a coherent readiness/image snapshot. It intentionally does393not read logs and therefore cannot provide log-loss evidence.394395The repository-wide AWS/EKS/O11y staging gate composes this secret-free mode396with AWS identity, EKS endpoint, auto-instrumentation, APM, and AWS integration397checks. See [`../../scripts/staging/README.md`](../../scripts/staging/README.md).398399## TA Completion Gate400401For TA/add-on work, also follow402[`../shared/ta_completion_gate.md`](../shared/ta_completion_gate.md). The403data ingest path must be configured and validated. Discover any404pre-built/package-shipped dashboards, then prove they are visible,405macro-aligned, and returning data. If the package ships no dashboards, record406that explicit package evidence. The currently audited TA source/package family407records no shipped `data/ui/views`; completion therefore depends on `_internal`408diagnostics and Observability/Platform telemetry rather than a nonexistent409packaged dashboard.410411Read `reference.md` for the option contract, lifecycle behavior, known product412documentation conflicts, and explicit handoffs.