Splunk Observability Isovalent Integration
Prerequisites
| Tool or access |
Purpose |
Verify |
| Bash and Python 3 |
Run bundled setup and validation helpers |
bash --version && python3 --version |
| Required product/platform access |
Inspect or configure the selected target |
Complete the documented preflight |
| Credential files for live modes |
Keep secrets out of chat |
Verify paths only |
Workflow Overview
┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘
When to Activate
- Wiring Cilium, Tetragon, or Hubble metrics into Splunk Observability Cloud, shipping Tetragon logs to Splunk
Platform, or validating Isovalent telemetry after platform install.
- Preview and review the splunk observability isovalent integration workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.
Scope
Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.
Examples
Inspect the supported setup modes before selecting one:
bash skills/splunk-observability-isovalent-integration/scripts/setup.sh --help
Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.
Inspect validation modes before running completion checks:
bash skills/splunk-observability-isovalent-integration/scripts/validate.sh --help
Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.
Troubleshooting
| Issue |
Cause |
Resolution |
| Preflight fails |
A required tool or access path is missing |
Resolve it before rendering or applying |
| Rendered assets are incomplete |
Required non-secret inputs are absent |
Complete intake and render again |
| Apply is blocked |
Review, credentials, or explicit acceptance is missing |
Use the documented handoff |
| Validation is incomplete |
Live evidence is unavailable |
Record the gap and keep completion open |
This skill wires an installed Isovalent stack to Splunk Observability Cloud and Splunk Platform. It depends on the platform install completed by cisco-isovalent-platform-setup. Run that first, then this.
What it renders
splunk-otel-overlay/values.overlay.yaml — Splunk OTel collector agent.config overlay with seven Prometheus scrape jobs and the filter/includemetrics allow-list. Designed to merge with the base values produced by splunk-observability-otel-collector-setup via yq deep-merge.
- Splunk Platform logs path (DEFAULT — file-based via OTel filelog receiver):
agent.extraVolumes + agent.extraVolumeMounts hostPath mount of /var/run/cilium/tetragon.
logsCollection.extraFileLogs.filelog/tetragon block with sourcetype cisco:isovalent, index cisco_isovalent.
splunkPlatform.logsEnabled: true.
- Coordinates with
cisco-isovalent-platform-setup's default Tetragon export.mode: file + exportDirectory: /var/run/cilium/tetragon.
- Alternative paths (behind explicit flags):
--export-mode stdout — Tetragon stdout + container log collection (no hostPath mount; useful when SCC/PSP blocks).
--legacy-fluentd-hec — fluentd splunk_hec block. DEPRECATED (fluent-plugin-splunk-hec archived 2025-06-24).
dashboards/cilium-by-isovalent.json and dashboards/hubble-by-isovalent.json — token-scrubbed re-exports (sourced from the Isovalent_Splunk_o11y reference repo's examples/*.json only after scripts/scrub-tokens.py confirms zero accessToken material).
detectors/*.yaml — starter detectors for cilium_*, hubble_*, tetragon_* series.
scripts/setup.sh, render_assets.py, validate.sh, handoff-base-collector.sh, handoff-hec-token.sh, handoff-cisco-security-cloud.sh, handoff-dashboards.sh, handoff-detectors.sh, scrub-tokens.py.
metadata.json.
Safety Rules
- Never ask for the Splunk Observability ingest token or Splunk Platform HEC token in conversation.
- File-backed token flags only:
--o11y-token-file (Splunk Observability Org access token; passed through to base collector).
--platform-hec-token-file (Splunk Platform HEC token; or --render-platform-hec-helper to delegate to splunk-hec-service-setup).
- Reject direct token flags (
--access-token, --token, --bearer-token, --api-token, --o11y-token, --sf-token, --platform-hec-token, --hec-token).
- Token files must be
chmod 600; the wrapper aborts otherwise (override with --allow-loose-token-perms, emits WARN). The actual apply happens via the rendered handoff-*.sh scripts, which inherit the same chmod 600 enforcement when they re-invoke the base collector / HEC / dashboard / native-ops setups.
- The renderer scrubs every dashboard JSON it ships against an
accessToken regex before writing. Render aborts if the source JSON contains plaintext token material.
Primary Workflow
Confirm the Isovalent stack is installed (run cisco-isovalent-platform-setup first).
Render:
bash skills/splunk-observability-isovalent-integration/scripts/setup.sh \
--render \
--validate \
--realm us0 \
--cluster-name lab-cluster \
--output-dir splunk-observability-isovalent-rendered
Review splunk-observability-isovalent-rendered/:
splunk-otel-overlay/values.overlay.yaml — the overlay; merge into the base collector values.
dashboards/*.json — token-scrubbed dashboard exports.
detectors/*.yaml — starter detectors.
scripts/handoff-*.sh — hand-off drivers for the four downstream skills.
Apply the overlay onto the existing Splunk OTel collector helm release
(recommended one-shot path). Merges this overlay onto current release
values via yq, normalizes legacy otlphttp exporter names to
otlp_http, preserves the live OBI eBPF ConfigMap data when OBI is
installed, runs helm upgrade --atomic --force-conflicts, and rolls the agent +
cluster-receiver. The helper preserves existing gateway/operator topology
unless collector.disable_gateway or collector.disable_operator is set,
auto-discovers the collector namespace, and pins the installed chart version
unless collector.namespace or collector.chart_version is set in the spec.
Refuses without
--accept-k8s-apply, refuses if no Cilium/Tetragon install is detected,
and prints the active kube-context first:
bash skills/splunk-observability-isovalent-integration/scripts/setup.sh \
--apply --accept-k8s-apply
--apply --accept-k8s-apply --dry-run runs helm upgrade --dry-run
without mutating the cluster.
Then provision HEC and route Tetragon logs to Splunk Platform via Cisco
Security Cloud:
bash splunk-observability-isovalent-rendered/scripts/handoff-hec-token.sh
bash splunk-observability-isovalent-rendered/scripts/handoff-cisco-security-cloud.sh
bash splunk-observability-isovalent-rendered/scripts/handoff-dashboards.sh
bash splunk-observability-isovalent-rendered/scripts/handoff-detectors.sh
Hand-offs
Out of scope
Validation
bash skills/splunk-observability-isovalent-integration/scripts/validate.sh
Static checks: overlay shape, token-scrub assertion, dashboard JSON validity, sourcetype/index match. With --live:
helm status for the OTel collector release.
- Pod-IP scrape probes for the seven Prometheus ports (uses
kubectl get --raw for Tetragon, NOT kubectl exec).
- Optional SignalFlow probe for
cilium_*, hubble_*, tetragon_* series presence.
- Optional Splunk Platform search check:
index=cisco_isovalent sourcetype=cisco:isovalent returns events.
See reference.md and the references/ annexes for collector-overlay details, Splunk Platform paths, Tetragon hostPath coordination, sourcetype reference, dashboards catalog, and troubleshooting.
1---2name: splunk-observability-isovalent-integration3description: "Use when wiring Cilium, Tetragon, or Hubble metrics into Splunk Observability Cloud, shipping Tetragon4 logs to Splunk Platform, or validating Isovalent telemetry after platform install. Wire an installed5 Isovalent stack (Cilium, Hubble, Tetragon, optional Hubble Enterprise or cilium-dnsproxy) to Splunk6 Observability Cloud and Splunk Platform. Renders Splunk OTel Collector scrape overlays, metric filters,7 Tetragon filelog ingestion defaults, stdout and legacy fluentd alternatives, dashboards, detectors, and8 handoff scripts for base collector, HEC, and Cisco Security Cloud ingestion."9compatibility: "No direct Splunk Platform runtime dependency. This workflow can be used alongside Splunk Cloud Platform 10.5.2605 through its documented external APIs or handoffs."10metadata:11 source: https://skillmd.com/skills/chambear2809/splunk-observability-isovalent-integration12 splunk_cloud_10_5: "not-applicable"13 compatibility_verified: "2026-08-20"14---1516# Splunk Observability Isovalent Integration1718## Prerequisites1920| Tool or access | Purpose | Verify |21|---|---|---|22| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |23| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |24| Credential files for live modes | Keep secrets out of chat | Verify paths only |2526## Workflow Overview2728```text29┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐30│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │31└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘32```3334## When to Activate3536- Wiring Cilium, Tetragon, or Hubble metrics into Splunk Observability Cloud, shipping Tetragon logs to Splunk37 Platform, or validating Isovalent telemetry after platform install.38- Preview and review the splunk observability isovalent integration workflow before any live apply phase.39- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.4041## Scope4243Follow the documented read-only or render-first path whenever it is available.44This skill does not imply permission to mutate live systems. Require explicit45apply flags, protected credentials, and operator review for state changes.4647## Examples4849Inspect the supported setup modes before selecting one:5051```bash52bash skills/splunk-observability-isovalent-integration/scripts/setup.sh --help53```5455Expected output: usage, supported modes, and required arguments are displayed56without changing the target environment.5758Inspect validation modes before running completion checks:5960```bash61bash skills/splunk-observability-isovalent-integration/scripts/validate.sh --help62```6364Expected output: offline, live, and completion options are displayed when the65skill supports them; help exits without mutation.6667## Troubleshooting6869| Issue | Cause | Resolution |70|---|---|---|71| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |72| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |73| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |74| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |7576This skill wires an installed Isovalent stack to Splunk Observability Cloud and Splunk Platform. It **depends on** the platform install completed by [cisco-isovalent-platform-setup](../cisco-isovalent-platform-setup/SKILL.md). Run that first, then this.7778## What it renders7980- `splunk-otel-overlay/values.overlay.yaml` — Splunk OTel collector agent.config overlay with seven Prometheus scrape jobs and the `filter/includemetrics` allow-list. Designed to merge with the base values produced by [splunk-observability-otel-collector-setup](../splunk-observability-otel-collector-setup/SKILL.md) via `yq` deep-merge.81- Splunk Platform logs path (DEFAULT — file-based via OTel filelog receiver):82 - `agent.extraVolumes` + `agent.extraVolumeMounts` hostPath mount of `/var/run/cilium/tetragon`.83 - `logsCollection.extraFileLogs.filelog/tetragon` block with sourcetype `cisco:isovalent`, index `cisco_isovalent`.84 - `splunkPlatform.logsEnabled: true`.85 - Coordinates with `cisco-isovalent-platform-setup`'s default Tetragon `export.mode: file` + `exportDirectory: /var/run/cilium/tetragon`.86- Alternative paths (behind explicit flags):87 - `--export-mode stdout` — Tetragon stdout + container log collection (no hostPath mount; useful when SCC/PSP blocks).88 - `--legacy-fluentd-hec` — fluentd `splunk_hec` block. **DEPRECATED** (`fluent-plugin-splunk-hec` archived 2025-06-24).89- `dashboards/cilium-by-isovalent.json` and `dashboards/hubble-by-isovalent.json` — token-scrubbed re-exports (sourced from the Isovalent_Splunk_o11y reference repo's `examples/*.json` only after `scripts/scrub-tokens.py` confirms zero `accessToken` material).90- `detectors/*.yaml` — starter detectors for `cilium_*`, `hubble_*`, `tetragon_*` series.91- `scripts/setup.sh`, `render_assets.py`, `validate.sh`, `handoff-base-collector.sh`, `handoff-hec-token.sh`, `handoff-cisco-security-cloud.sh`, `handoff-dashboards.sh`, `handoff-detectors.sh`, `scrub-tokens.py`.92- `metadata.json`.9394## Safety Rules9596- Never ask for the Splunk Observability ingest token or Splunk Platform HEC token in conversation.97- File-backed token flags only:98 - `--o11y-token-file` (Splunk Observability Org access token; passed through to base collector).99 - `--platform-hec-token-file` (Splunk Platform HEC token; or `--render-platform-hec-helper` to delegate to `splunk-hec-service-setup`).100- Reject direct token flags (`--access-token`, `--token`, `--bearer-token`, `--api-token`, `--o11y-token`, `--sf-token`, `--platform-hec-token`, `--hec-token`).101- Token files must be `chmod 600`; the wrapper aborts otherwise (override with `--allow-loose-token-perms`, emits WARN). The actual apply happens via the rendered handoff-*.sh scripts, which inherit the same chmod 600 enforcement when they re-invoke the base collector / HEC / dashboard / native-ops setups.102- The renderer scrubs every dashboard JSON it ships against an `accessToken` regex before writing. Render aborts if the source JSON contains plaintext token material.103104## Primary Workflow1051061. Confirm the Isovalent stack is installed (run `cisco-isovalent-platform-setup` first).1071082. Render:109110 ```bash111 bash skills/splunk-observability-isovalent-integration/scripts/setup.sh \112 --render \113 --validate \114 --realm us0 \115 --cluster-name lab-cluster \116 --output-dir splunk-observability-isovalent-rendered117 ```1181193. Review `splunk-observability-isovalent-rendered/`:120 - `splunk-otel-overlay/values.overlay.yaml` — the overlay; merge into the base collector values.121 - `dashboards/*.json` — token-scrubbed dashboard exports.122 - `detectors/*.yaml` — starter detectors.123 - `scripts/handoff-*.sh` — hand-off drivers for the four downstream skills.1241254. Apply the overlay onto the existing Splunk OTel collector helm release126 (recommended one-shot path). Merges this overlay onto current release127 values via `yq`, normalizes legacy `otlphttp` exporter names to128 `otlp_http`, preserves the live OBI eBPF ConfigMap data when OBI is129 installed, runs `helm upgrade --atomic --force-conflicts`, and rolls the agent +130 cluster-receiver. The helper preserves existing gateway/operator topology131 unless `collector.disable_gateway` or `collector.disable_operator` is set,132 auto-discovers the collector namespace, and pins the installed chart version133 unless `collector.namespace` or `collector.chart_version` is set in the spec.134 Refuses without135 `--accept-k8s-apply`, refuses if no Cilium/Tetragon install is detected,136 and prints the active kube-context first:137138 ```bash139 bash skills/splunk-observability-isovalent-integration/scripts/setup.sh \140 --apply --accept-k8s-apply141 ```142143 `--apply --accept-k8s-apply --dry-run` runs `helm upgrade --dry-run`144 without mutating the cluster.145146 Then provision HEC and route Tetragon logs to Splunk Platform via Cisco147 Security Cloud:148149 ```bash150 bash splunk-observability-isovalent-rendered/scripts/handoff-hec-token.sh151 bash splunk-observability-isovalent-rendered/scripts/handoff-cisco-security-cloud.sh152 bash splunk-observability-isovalent-rendered/scripts/handoff-dashboards.sh153 bash splunk-observability-isovalent-rendered/scripts/handoff-detectors.sh154 ```155156## Hand-offs157158- Splunk OTel Collector base install: [splunk-observability-otel-collector-setup](../splunk-observability-otel-collector-setup/SKILL.md). Render the base values, then merge our overlay via `yq` deep-merge.159- Splunk Platform HEC token: [splunk-hec-service-setup](../splunk-hec-service-setup/SKILL.md).160- Splunk Platform Tetragon log ingestion: [cisco-security-cloud-setup](../cisco-security-cloud-setup/SKILL.md) with `PRODUCT=isovalent` (sourcetype `cisco:isovalent:processExec`, index `cisco_isovalent`); confirmed in [skills/cisco-security-cloud-setup/products.json](../cisco-security-cloud-setup/products.json) lines 200-219.161- Dashboards: [splunk-observability-dashboard-builder](../splunk-observability-dashboard-builder/SKILL.md).162- Detectors: [splunk-observability-native-ops](../splunk-observability-native-ops/SKILL.md).163164## Out of scope165166- Cilium / Tetragon / Hubble install lifecycle — handled by [cisco-isovalent-platform-setup](../cisco-isovalent-platform-setup/SKILL.md).167- Splunk OTel collector base install — handled by [splunk-observability-otel-collector-setup](../splunk-observability-otel-collector-setup/SKILL.md).168169## Validation170171```bash172bash skills/splunk-observability-isovalent-integration/scripts/validate.sh173```174175Static checks: overlay shape, token-scrub assertion, dashboard JSON validity, sourcetype/index match. With `--live`:176177- `helm status` for the OTel collector release.178- Pod-IP scrape probes for the seven Prometheus ports (uses `kubectl get --raw` for Tetragon, NOT `kubectl exec`).179- Optional SignalFlow probe for `cilium_*`, `hubble_*`, `tetragon_*` series presence.180- Optional Splunk Platform search check: `index=cisco_isovalent sourcetype=cisco:isovalent` returns events.181182See `reference.md` and the `references/` annexes for collector-overlay details, Splunk Platform paths, Tetragon hostPath coordination, sourcetype reference, dashboards catalog, and troubleshooting.