Splunk Observability Cisco AI Pod Integration (Umbrella)
Prerequisites
| Tool or access |
Purpose |
Verify |
| Bash and Python 3 |
Run bundled setup and validation helpers |
bash --version && python3 --version |
| Required product/platform access |
Inspect or configure the selected target |
Complete the documented preflight |
| Credential files for live modes |
Keep secrets out of chat |
Verify paths only |
Workflow Overview
┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘
When to Activate
- Deploying Splunk Observability Cloud for a Cisco AI Pod with UCS, Nexus, NVIDIA GPUs, NIM/vLLM inference, and
storage telemetry. Hand off base collector, HEC, dashboards, and detectors to the owning skills.
- Preview and review the splunk observability cisco ai pod integration workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.
Scope
Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.
Examples
Inspect the supported setup modes before selecting one:
bash skills/splunk-observability-cisco-ai-pod-integration/scripts/setup.sh --help
Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.
Inspect validation modes before running completion checks:
bash skills/splunk-observability-cisco-ai-pod-integration/scripts/validate.sh --help
Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.
Troubleshooting
| Issue |
Cause |
Resolution |
| Preflight fails |
A required tool or access path is missing |
Resolve it before rendering or applying |
| Rendered assets are incomplete |
Required non-secret inputs are absent |
Complete intake and render again |
| Apply is blocked |
Review, credentials, or explicit acceptance is missing |
Use the documented handoff |
| Validation is incomplete |
Live evidence is unavailable |
Record the gap and keep completion open |
This is the AI Pod umbrella that ties together every component skill needed for end-to-end Cisco AI Pod observability in Splunk Observability Cloud. It composes:
- splunk-observability-cisco-nexus-integration for Cisco Nexus 9000 fabric metrics (cisco_os receiver).
- splunk-observability-cisco-intersight-integration for Cisco UCS metrics via Intersight OTel deployment.
- splunk-observability-nvidia-gpu-integration for NVIDIA GPU telemetry via DCGM Exporter.
And adds AI-Pod-specific bits documented in the configuration guide and production-validated by an OpenShift deployment:
- NIM scrapes (multi-job: llm/embedqa/rerankqa, port 8000
/v1/metrics).
- vLLM scrape (port 8000
/metrics).
- Milvus vector DB scrape (port 9091).
- NetApp Trident storage scrape (port 8001
/metrics).
- Pure Portworx storage scrape (ports 17001 + 17018).
- Redfish exporter (user-supplied) on port 9210.
- Cisco AI PODs Splunk Observability dashboard pipeline (
metrics/cisco-ai-pods, unfiltered).
- NIM dashboard pipeline (
metrics/nvidianim-metrics, unfiltered).
k8s_attributes/nim processor for app -> model_name extraction.
- OpenShift SCC helper,
workshop/multi-tenant.sh, and dual-pipeline filtering pattern.
Critical production lessons encoded
These are the silent failure traps the umbrella prevents:
- RBAC gap: base chart's ClusterRole grants only
pods and services. Any kubernetes_sd_configs.role: endpoints scrape (e.g. NIM in endpoint mode) silently fails with endpoints is forbidden. The umbrella emits the rbac.customRules block with endpoints + discovery.k8s.io/endpointslices get/list/watch when needed.
- receiver_creator naming:
receiver_creator/dcgm-cisco, NOT receiver_creator/nvidia (collision with chart autodetect). Inherited from the GPU child skill.
- DCGM dual-label discovery: matches both
app and app.kubernetes.io/name. Inherited from the GPU child skill.
- Dual-pipeline filtering: filtered standard pipeline + unfiltered specialized pipelines for AI Pod dashboards. Smarter than the canonical single-pipeline pattern.
- OpenShift defaults:
kubeletstats.insecure_skip_verify: true (REQUIRED), certmanager.enabled: false, cloudProvider: "".
- Existing collector apply: use
--apply-existing-collector when a Splunk OTel Collector is already running. This path renders the overlay, reads current Helm values without persisting the token, removes stale receiver_creator/nvidia, wires otlp into the metrics pipeline for Intersight, applies via Helm, restarts the existing collector agent, restarts Intersight, and runs live validation.
- Helm token pattern: apply scripts use a file-backed token (
--set-file splunkObservability.accessToken=...) so the token is never written to a tracked values file or temporary values file.
Composition model
When you run --render, the umbrella:
- Invokes each child skill's renderer to produce its overlay under a sub-directory.
- Merges the child overlays into a unified
splunk-otel-overlay/values.overlay.yaml with the renderer's deterministic Python deep-merge. The rendered base-collector handoff uses yq later to merge that reviewed overlay with base collector values.
- Adds AI-Pod-specific blocks on top of the merged overlay.
- Renders unified handoff scripts.
When you run --apply-existing-collector, the umbrella applies its rendered overlay to the already running Splunk OTel Collector Helm release instead of standing up a second collector.
What it renders (composed + AI-Pod-specific)
splunk-otel-overlay/values.overlay.yaml — composed overlay (Nexus + Intersight + GPU children + AI-Pod additions).
child-renders/<skill>/ — each child skill's full rendered output (preserved for debugging the merge).
intersight-integration/ — from the Intersight child.
secrets/cisco-nexus-ssh-secret.yaml — from the Nexus child.
dcgm-pod-labels-patch/ — from the GPU child when --enable-dcgm-pod-labels.
- NIM, vLLM, Milvus, Trident, Portworx, and Redfish scrape configuration embedded in
splunk-otel-overlay/values.overlay.yaml.
openshift/scc.sh — OpenShift SCC helper script.
workshop/multi-tenant.sh — Workshop multi-tenant deploy script (when --workshop-mode).
dashboards/ — AI-Pod-specific dashboards for NIM/vLLM inference, Milvus, and Trident/Portworx storage.
detectors/ — AI-Pod-specific detectors (vLLM error rate, NIM TTFT regression, Milvus query latency, Portworx node offline, Trident volume allocation).
scripts/handoff-base-collector.sh — emits the base collector + merge command with --distribution openshift (default).
scripts/handoff-hec-token.sh — for K8s container log shipping to Splunk Platform.
scripts/handoff-dashboards.sh, handoff-detectors.sh — emit reviewed dashboard and detector commands across all four skills.
scripts/explain-composition.sh — prints the per-child contribution summary.
metadata.json.
Safety Rules
- File-backed token flags only:
--o11y-token-file (Splunk Observability Org access token; passed through to all child skills + base collector).
--platform-hec-token-file (optional; for K8s container logs to Splunk Platform).
--intersight-key-id-file and --intersight-key-file (passed through to the Intersight child).
- Reject every direct token / key flag.
- Token files must be
chmod 600; --allow-loose-token-perms overrides with WARN.
- Cisco Nexus SSH credentials handled by the Nexus child (K8s Secret stub; user creates the Secret out-of-band).
Primary Workflow
Confirm prerequisites are installed: NVIDIA GPU Operator (or standalone DCGM Exporter), NIM/vLLM with the standard pod labels, Milvus, NetApp Trident, Pure Portworx, Redfish exporter, Cisco Intersight account + API key.
Render the composed overlay:
bash skills/splunk-observability-cisco-ai-pod-integration/scripts/setup.sh \
--render --validate \
--realm us0 \
--cluster-name atl-ai-pod \
--distribution openshift \
--nim-scrape-mode endpoints \
--enable-dcgm-pod-labels \
--output-dir splunk-observability-cisco-ai-pod-rendered
If a Splunk OTel Collector is already running, apply the overlay in place and run live validation:
bash skills/splunk-observability-cisco-ai-pod-integration/scripts/setup.sh \
--render --apply-existing-collector --validate --live \
--realm us0 \
--cluster-name atl-ai-pod \
--distribution openshift \
--collector-release splunk-otel-collector \
--collector-namespace splunk-otel \
--o11y-token-file /path/to/o11y-token \
--output-dir splunk-observability-cisco-ai-pod-rendered
For greenfield installs, apply child manifests (Intersight, optional DCGM patch) + merge overlay + apply via base collector:
bash splunk-observability-cisco-ai-pod-rendered/scripts/handoff-base-collector.sh
bash splunk-observability-cisco-ai-pod-rendered/scripts/handoff-dashboards.sh
bash splunk-observability-cisco-ai-pod-rendered/scripts/handoff-detectors.sh
Hand-offs
Out of scope
- All children's out-of-scope items (NVIDIA GPU Operator install, DCGM Exporter install, NIM/vLLM/Milvus/Trident/Portworx/Redfish exporter deployment, OpenShift cluster bootstrap, Cisco Intersight account creation).
Validation
bash skills/splunk-observability-cisco-ai-pod-integration/scripts/validate.sh
Runs each child skill's validate.sh recursively, then checks the composed overlay, endpoint-discovery RBAC, OpenShift kubelet settings, and rendered secret safety. With --live, it probes collector and Intersight resources and logs plus the live collector ConfigMap. The umbrella validator does not make direct SignalFlow API probes for NIM, Milvus, or vLLM metrics.
With --live, validation prefers oc, falls back to kubectl, passes --live through to child validators, and fails on Intersight OTLP export errors such as unknown service opentelemetry.proto.collector.metrics.v1.MetricsService.
See reference.md and references/composition-and-overlay-merge.md, nim-vllm-scrape-catalog.md, milvus-storage-redfish.md, openshift-scc.md, workshop-multi-tenant.md, ai-pod-dashboards-catalog.md, endpoints-rbac-patch.md, dual-pipeline-filtering.md, nim-scrape-modes.md, production-troubleshooting-reference.md, troubleshooting.md for the full annexes.
1---2name: splunk-observability-cisco-ai-pod-integration3description: "Use when deploying Splunk Observability Cloud for a Cisco AI Pod with UCS, Nexus, NVIDIA GPUs, NIM/vLLM4 inference, and storage telemetry. Hand off base collector, HEC, dashboards, and detectors to the owning5 skills. Compose Cisco Nexus, Cisco Intersight, and NVIDIA GPU Observability skills into a Cisco AI Pod6 overlay, then add NIM, vLLM, Milvus, NetApp Trident, Pure Portworx, Redfish exporter, OpenShift SCC,7 workshop tenancy, RBAC, receiver naming, DCGM discovery, dual-pipeline filtering, NIM model-name8 extraction, and existing-collector cleanup patterns."9compatibility: "No direct Splunk Platform runtime dependency. This workflow can be used alongside Splunk Cloud Platform 10.5.2605 through its documented external APIs or handoffs."10metadata:11 source: https://skillmd.com/skills/chambear2809/splunk-observability-cisco-ai-pod-integration12 splunk_cloud_10_5: "not-applicable"13 compatibility_verified: "2026-08-20"14---1516# Splunk Observability Cisco AI Pod Integration (Umbrella)1718## Prerequisites1920| Tool or access | Purpose | Verify |21|---|---|---|22| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |23| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |24| Credential files for live modes | Keep secrets out of chat | Verify paths only |2526## Workflow Overview2728```text29┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐30│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │31└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘32```3334## When to Activate3536- Deploying Splunk Observability Cloud for a Cisco AI Pod with UCS, Nexus, NVIDIA GPUs, NIM/vLLM inference, and37 storage telemetry. Hand off base collector, HEC, dashboards, and detectors to the owning skills.38- Preview and review the splunk observability cisco ai pod integration workflow before any live apply phase.39- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.4041## Scope4243Follow the documented read-only or render-first path whenever it is available.44This skill does not imply permission to mutate live systems. Require explicit45apply flags, protected credentials, and operator review for state changes.4647## Examples4849Inspect the supported setup modes before selecting one:5051```bash52bash skills/splunk-observability-cisco-ai-pod-integration/scripts/setup.sh --help53```5455Expected output: usage, supported modes, and required arguments are displayed56without changing the target environment.5758Inspect validation modes before running completion checks:5960```bash61bash skills/splunk-observability-cisco-ai-pod-integration/scripts/validate.sh --help62```6364Expected output: offline, live, and completion options are displayed when the65skill supports them; help exits without mutation.6667## Troubleshooting6869| Issue | Cause | Resolution |70|---|---|---|71| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |72| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |73| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |74| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |7576This is the **AI Pod umbrella** that ties together every component skill needed for end-to-end Cisco AI Pod observability in Splunk Observability Cloud. It composes:77781. [splunk-observability-cisco-nexus-integration](../splunk-observability-cisco-nexus-integration/SKILL.md) for Cisco Nexus 9000 fabric metrics (cisco_os receiver).792. [splunk-observability-cisco-intersight-integration](../splunk-observability-cisco-intersight-integration/SKILL.md) for Cisco UCS metrics via Intersight OTel deployment.803. [splunk-observability-nvidia-gpu-integration](../splunk-observability-nvidia-gpu-integration/SKILL.md) for NVIDIA GPU telemetry via DCGM Exporter.8182And adds **AI-Pod-specific bits** documented in the configuration guide and production-validated by an OpenShift deployment:8384- NIM scrapes (multi-job: llm/embedqa/rerankqa, port 8000 `/v1/metrics`).85- vLLM scrape (port 8000 `/metrics`).86- Milvus vector DB scrape (port 9091).87- NetApp Trident storage scrape (port 8001 `/metrics`).88- Pure Portworx storage scrape (ports 17001 + 17018).89- Redfish exporter (user-supplied) on port 9210.90- Cisco AI PODs Splunk Observability dashboard pipeline (`metrics/cisco-ai-pods`, unfiltered).91- NIM dashboard pipeline (`metrics/nvidianim-metrics`, unfiltered).92- `k8s_attributes/nim` processor for `app -> model_name` extraction.93- OpenShift SCC helper, `workshop/multi-tenant.sh`, and dual-pipeline filtering pattern.9495## Critical production lessons encoded9697These are the **silent failure traps** the umbrella prevents:98991. **RBAC gap**: base chart's ClusterRole grants only `pods` and `services`. Any `kubernetes_sd_configs.role: endpoints` scrape (e.g. NIM in endpoint mode) silently fails with `endpoints is forbidden`. The umbrella emits the `rbac.customRules` block with `endpoints` + `discovery.k8s.io/endpointslices` get/list/watch when needed.1002. **receiver_creator naming**: `receiver_creator/dcgm-cisco`, NOT `receiver_creator/nvidia` (collision with chart autodetect). Inherited from the GPU child skill.1013. **DCGM dual-label discovery**: matches both `app` and `app.kubernetes.io/name`. Inherited from the GPU child skill.1024. **Dual-pipeline filtering**: filtered standard pipeline + unfiltered specialized pipelines for AI Pod dashboards. Smarter than the canonical single-pipeline pattern.1035. **OpenShift defaults**: `kubeletstats.insecure_skip_verify: true` (REQUIRED), `certmanager.enabled: false`, `cloudProvider: ""`.1046. **Existing collector apply**: use `--apply-existing-collector` when a Splunk OTel Collector is already running. This path renders the overlay, reads current Helm values without persisting the token, removes stale `receiver_creator/nvidia`, wires `otlp` into the metrics pipeline for Intersight, applies via Helm, restarts the existing collector agent, restarts Intersight, and runs live validation.1057. **Helm token pattern**: apply scripts use a file-backed token (`--set-file splunkObservability.accessToken=...`) so the token is never written to a tracked values file or temporary values file.106107## Composition model108109When you run `--render`, the umbrella:1101111. Invokes each child skill's renderer to produce its overlay under a sub-directory.1122. Merges the child overlays into a unified `splunk-otel-overlay/values.overlay.yaml` with the renderer's deterministic Python deep-merge. The rendered base-collector handoff uses `yq` later to merge that reviewed overlay with base collector values.1133. Adds AI-Pod-specific blocks on top of the merged overlay.1144. Renders unified handoff scripts.115116When you run `--apply-existing-collector`, the umbrella applies its rendered overlay to the already running Splunk OTel Collector Helm release instead of standing up a second collector.117118## What it renders (composed + AI-Pod-specific)119120- `splunk-otel-overlay/values.overlay.yaml` — composed overlay (Nexus + Intersight + GPU children + AI-Pod additions).121- `child-renders/<skill>/` — each child skill's full rendered output (preserved for debugging the merge).122- `intersight-integration/` — from the Intersight child.123- `secrets/cisco-nexus-ssh-secret.yaml` — from the Nexus child.124- `dcgm-pod-labels-patch/` — from the GPU child when `--enable-dcgm-pod-labels`.125- NIM, vLLM, Milvus, Trident, Portworx, and Redfish scrape configuration embedded in `splunk-otel-overlay/values.overlay.yaml`.126- `openshift/scc.sh` — OpenShift SCC helper script.127- `workshop/multi-tenant.sh` — Workshop multi-tenant deploy script (when `--workshop-mode`).128- `dashboards/` — AI-Pod-specific dashboards for NIM/vLLM inference, Milvus, and Trident/Portworx storage.129- `detectors/` — AI-Pod-specific detectors (vLLM error rate, NIM TTFT regression, Milvus query latency, Portworx node offline, Trident volume allocation).130- `scripts/handoff-base-collector.sh` — emits the base collector + merge command with `--distribution openshift` (default).131- `scripts/handoff-hec-token.sh` — for K8s container log shipping to Splunk Platform.132- `scripts/handoff-dashboards.sh`, `handoff-detectors.sh` — emit reviewed dashboard and detector commands across all four skills.133- `scripts/explain-composition.sh` — prints the per-child contribution summary.134- `metadata.json`.135136## Safety Rules137138- File-backed token flags only:139 - `--o11y-token-file` (Splunk Observability Org access token; passed through to all child skills + base collector).140 - `--platform-hec-token-file` (optional; for K8s container logs to Splunk Platform).141 - `--intersight-key-id-file` and `--intersight-key-file` (passed through to the Intersight child).142- Reject every direct token / key flag.143- Token files must be `chmod 600`; `--allow-loose-token-perms` overrides with WARN.144- Cisco Nexus SSH credentials handled by the Nexus child (K8s Secret stub; user creates the Secret out-of-band).145146## Primary Workflow1471481. Confirm prerequisites are installed: NVIDIA GPU Operator (or standalone DCGM Exporter), NIM/vLLM with the standard pod labels, Milvus, NetApp Trident, Pure Portworx, Redfish exporter, Cisco Intersight account + API key.1491502. Render the composed overlay:151152 ```bash153 bash skills/splunk-observability-cisco-ai-pod-integration/scripts/setup.sh \154 --render --validate \155 --realm us0 \156 --cluster-name atl-ai-pod \157 --distribution openshift \158 --nim-scrape-mode endpoints \159 --enable-dcgm-pod-labels \160 --output-dir splunk-observability-cisco-ai-pod-rendered161 ```1621633. If a Splunk OTel Collector is already running, apply the overlay in place and run live validation:164165 ```bash166 bash skills/splunk-observability-cisco-ai-pod-integration/scripts/setup.sh \167 --render --apply-existing-collector --validate --live \168 --realm us0 \169 --cluster-name atl-ai-pod \170 --distribution openshift \171 --collector-release splunk-otel-collector \172 --collector-namespace splunk-otel \173 --o11y-token-file /path/to/o11y-token \174 --output-dir splunk-observability-cisco-ai-pod-rendered175 ```1761774. For greenfield installs, apply child manifests (Intersight, optional DCGM patch) + merge overlay + apply via base collector:178179 ```bash180 bash splunk-observability-cisco-ai-pod-rendered/scripts/handoff-base-collector.sh181 bash splunk-observability-cisco-ai-pod-rendered/scripts/handoff-dashboards.sh182 bash splunk-observability-cisco-ai-pod-rendered/scripts/handoff-detectors.sh183 ```184185## Hand-offs186187- Splunk OTel Collector base install: [splunk-observability-otel-collector-setup](../splunk-observability-otel-collector-setup/SKILL.md) with `--distribution openshift` (default; configurable).188- HEC for K8s container logs: [splunk-hec-service-setup](../splunk-hec-service-setup/SKILL.md).189- Dashboards: [splunk-observability-dashboard-builder](../splunk-observability-dashboard-builder/SKILL.md).190- Detectors: [splunk-observability-native-ops](../splunk-observability-native-ops/SKILL.md).191- Component skills (composed): Nexus / Intersight / GPU child skills.192193## Out of scope194195- All children's out-of-scope items (NVIDIA GPU Operator install, DCGM Exporter install, NIM/vLLM/Milvus/Trident/Portworx/Redfish exporter deployment, OpenShift cluster bootstrap, Cisco Intersight account creation).196197## Validation198199```bash200bash skills/splunk-observability-cisco-ai-pod-integration/scripts/validate.sh201```202203Runs each child skill's `validate.sh` recursively, then checks the composed overlay, endpoint-discovery RBAC, OpenShift kubelet settings, and rendered secret safety. With `--live`, it probes collector and Intersight resources and logs plus the live collector ConfigMap. The umbrella validator does not make direct SignalFlow API probes for NIM, Milvus, or vLLM metrics.204205With `--live`, validation prefers `oc`, falls back to `kubectl`, passes `--live` through to child validators, and fails on Intersight OTLP export errors such as `unknown service opentelemetry.proto.collector.metrics.v1.MetricsService`.206207See `reference.md` and `references/composition-and-overlay-merge.md`, `nim-vllm-scrape-catalog.md`, `milvus-storage-redfish.md`, `openshift-scc.md`, `workshop-multi-tenant.md`, `ai-pod-dashboards-catalog.md`, `endpoints-rbac-patch.md`, `dual-pipeline-filtering.md`, `nim-scrape-modes.md`, `production-troubleshooting-reference.md`, `troubleshooting.md` for the full annexes.