Splunk Observability Cisco Nexus Integration

Splunk Observability Cisco Nexus Integration

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Send Cisco Nexus, NX-OS, IOS-XE, or IOS-XR device metrics to Splunk Observability Cloud, configure the cisco_os receiver, set up multi-device Nexus telemetry, or render dashboards/detectors for Cisco data center fabric.
• Preview and review the splunk observability cisco nexus integration workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-cisco-nexus-integration/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-cisco-nexus-integration/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

This is a standalone reusable skill for Cisco Nexus 9000 (and any cisco_os-receiver-supported device) metrics in Splunk Observability Cloud. It is independent of the AI Pod umbrella — useful for any data center with Nexus fabric. The AI Pod skill composes this skill via subprocess + yq deep-merge.

The Splunk Platform TA path for Nexus / ACI / Nexus Dashboard lives in cisco-dc-networking-setup. That's a different layer (Splunk Platform side); this skill is the O11y side.

What it renders

• splunk-otel-overlay/values.overlay.yaml — clusterReceiver.config.receivers.cisco_os block in the new multi-device + global-scrapers format (cisco_os receiver in upstream contrib v0.149.0+). Devices reference K8s Secret-mounted creds; supports password or key_file per device. Scrapers: system (cisco.device.up, system.cpu.utilization, system.memory.utilization) and interfaces (system.network.io, system.network.errors, system.network.packet.dropped|count, system.network.interface.status).
• splunk-otel-overlay/cisco-os-pipeline.yaml — metrics/cisco-os-metrics pipeline with signalfx exporter and memory_limiter|batch|resourcedetection|resource processors.
• secrets/cisco-nexus-ssh-secret.yaml — K8s Secret manifest stub (rendered with placeholders; user creates with kubectl create secret generic --from-file=...). Renderer never reads SSH passwords.
• dashboards/<name>.signalflow.yaml — Nexus port utilization, packet errors, drop rates, system CPU/memory, interface status.
• detectors/<name>.yaml — interface down, packet drop rate threshold, memory pressure.
• scripts/setup.sh, render_assets.py, validate.sh, handoff-base-collector.sh, handoff-dashboards.sh, handoff-detectors.sh.
• metadata.json.

Safety Rules

• Never ask for Cisco Nexus SSH passwords or SSH keys in conversation.
• The renderer writes a K8s Secret manifest stub with placeholder values; the operator creates the actual Secret out-of-band with kubectl create secret generic --from-file=....
• --o11y-token-file flag is for the Splunk Observability Org access token (passed through to base collector). Reject --o11y-token, --access-token, --token, --bearer-token, --api-token, --sf-token.
• Token files must be chmod 600; --allow-loose-token-perms overrides with WARN.

Primary Workflow

1. Identify your Nexus devices (hostnames or management IPs) and gather per-device SSH credentials (out-of-band).

2. Render:

   bash skills/splunk-observability-cisco-nexus-integration/scripts/setup.sh \
     --render --validate \
     --realm us0 \
     --cluster-name lab-cluster \
     --nexus-device "core-switch-01:192.168.1.10" \
     --nexus-device "core-switch-02:192.168.1.11" \
     --output-dir splunk-observability-cisco-nexus-rendered
   

3. Review splunk-observability-cisco-nexus-rendered/ and create the SSH credentials Secret:

   kubectl create secret generic cisco-nexus-ssh \
     --from-literal=username=splunk-otel \
     --from-file=password=/tmp/nexus_password \
     -n splunk-otel
   

4. Apply directly via the skill (recommended). This merges the rendered overlay onto the existing Splunk OTel collector helm release values and runs helm upgrade --atomic. Refuses without --accept-k8s-apply, refuses if the cisco-nexus-ssh Secret from step 3 is missing, and prints the active kube-context first:

   bash skills/splunk-observability-cisco-nexus-integration/scripts/setup.sh \
     --apply --accept-k8s-apply \
     --realm us0 --cluster-name lab-cluster \
     --nexus-device "core-switch-01:192.168.1.10" \
     --nexus-device "core-switch-02:192.168.1.11"
   

   --apply --accept-k8s-apply --dry-run runs helm upgrade --dry-run without mutating the cluster.

   For dashboards / detectors, the rendered handoff scripts call into the owning skills:

   bash splunk-observability-cisco-nexus-rendered/scripts/handoff-dashboards.sh
   bash splunk-observability-cisco-nexus-rendered/scripts/handoff-detectors.sh
   

Hand-offs

• Splunk OTel Collector base install: splunk-observability-otel-collector-setup.
• Dashboards: splunk-observability-dashboard-builder.
• Detectors: splunk-observability-native-ops.

Out of scope (companion skills)

• Splunk Platform TA path for Nexus / ACI / Nexus Dashboard: cisco-dc-networking-setup.
• Cisco Catalyst Center / ISE / SD-WAN / Cyber Vision: cisco-catalyst-ta-setup.

Validation

bash skills/splunk-observability-cisco-nexus-integration/scripts/validate.sh

Static checks: overlay shape, Secret manifest placeholder validity, no inline credentials. With --live: helm status, OTel collector pod logs grep for cisco_os scrape errors.

See reference.md and the references/ annexes for the cisco_os receiver schema, multi-device config, SSH secrets, dashboards catalog, and troubleshooting.