Splunk Observability AI Agent Monitoring Setup
Prerequisites
| Tool or access |
Purpose |
Verify |
| Bash and Python 3 |
Run bundled setup and validation helpers |
bash --version && python3 --version |
| Required product/platform access |
Inspect or configure the selected target |
Complete the documented preflight |
| Credential files for live modes |
Keep secrets out of chat |
Verify paths only |
Workflow Overview
┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘
When to Activate
- Setting up or auditing Splunk AI Agent Monitoring, GenAI telemetry packages, AI agent evaluation telemetry, or
adjacent AI infrastructure observability.
- Preview and review the splunk observability ai agent monitoring setup workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.
Scope
Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.
Examples
Inspect the supported setup modes before selecting one:
bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/setup.sh --help
Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.
Inspect validation modes before running completion checks:
bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/validate.sh --help
Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.
Troubleshooting
| Issue |
Cause |
Resolution |
| Preflight fails |
A required tool or access path is missing |
Resolve it before rendering or applying |
| Rendered assets are incomplete |
Required non-secret inputs are absent |
Complete intake and render again |
| Apply is blocked |
Review, credentials, or explicit acceptance is missing |
Use the documented handoff |
| Validation is incomplete |
Live evidence is unavailable |
Record the gap and keep completion open |
Overview
Use this skill to build a render-first setup plan for Splunk AI Agent Monitoring and adjacent AI Infrastructure Monitoring. The skill owns orchestration, validation, coverage reporting, and safe delegation; it does not mark UI-only workflows as API-applied.
The renderer always emits a coverage report. Treat coverage-report.json as the source of truth for what is delegated_apply, render, deeplink, handoff, or not_applicable.
Safety Rules
- Never ask for Splunk Observability tokens, Splunk Platform passwords, HEC tokens, LLM API keys, or client secrets in conversation.
- Never pass secrets directly on the command line or as environment-variable prefixes.
- Use file-based secrets only, such as
SPLUNK_O11Y_TOKEN_FILE, --o11y-token-file, --platform-hec-token-file, or the delegated skill's file flags.
- Default prompt and response content capture to off.
- Require explicit accept flags before rendering content capture or LLM-as-judge evaluation env vars.
- Do not promise PII redaction, hashing, or truncation. The upstream GenAI utility marks those helpers as not implemented.
Primary Workflow
Start from template.example, or create a small JSON/YAML spec with the target realm, collector mode, agent frameworks, and AI infrastructure products.
For Kubernetes runtime handoffs, set deployment.workload_kind, deployment.workload_namespace, deployment.workload_name, and deployment.container_name to the exact application workload and container that should receive the environment variables.
Render and validate:
bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/setup.sh \
--render \
--validate \
--spec skills/splunk-observability-ai-agent-monitoring-setup/template.example \
--output-dir splunk-observability-ai-agent-monitoring-rendered
Review the generated files:
coverage-report.json and coverage-report.md
apply-plan.json
runtime/python.env and runtime/requirements.txt
collector/values-ai-agent-monitoring.yaml
handoff.md and doctor-report.md
Apply only when explicitly requested:
bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/setup.sh \
--apply collector,hec,loc \
--spec my-ai-agent-monitoring.json \
--o11y-token-file /tmp/splunk_o11y_token
Apply Model
Supported --apply sections are:
collector delegates to splunk-observability-otel-collector-setup using --apply-k8s or --apply-linux.
hec delegates to splunk-hec-service-setup --phase apply.
loc delegates to splunk-observability-cloud-integration-setup --apply log_observer_connect; the Observability UI connection wizard remains deeplink.
python-runtime and kubernetes-runtime render deterministic application changes and print operator handoffs because the skill does not own the user's application repository or cluster workloads.
ai-infra-collector delegates to existing product skills when available, such as Cisco AI PODs and NVIDIA GPU, otherwise renders collector overlay handoffs.
dashboards delegates to splunk-observability-dashboard-builder.
detectors delegates to splunk-observability-native-ops.
Validation Rules
Validation must fail for:
send_otlp_histograms: false, because the Agents page requires histogram metrics.
- Prompt/response content capture without
--accept-content-capture or accept_content_capture: true.
- Instrumentation-side evaluations without
--accept-evaluation-cost or accept_evaluation_cost: true.
- OpenAI evaluations without
OTEL_INSTRUMENTATION_GENAI_EVALS_SEPARATE_PROCESS=true.
- Unsupported package names such as
splunk-otel-instrumentation-openai-v2 or splunk-otel-instrumentation-vertexai.
- Selected packages whose Python requirement exceeds the declared runtime version.
References
- Read
reference.md for the CLI and generated artifact contract.
- Read
references/coverage.md when changing coverage status or adding product support.
- Read
references/package-catalog.md before changing package names, Python floors, or framework mappings.
- Read
references/apply-and-safety.md before changing live apply behavior.
- Read
references/troubleshooting.md when updating doctor output.
1---2name: splunk-observability-ai-agent-monitoring-setup3description: "Use when setting up or auditing Splunk AI Agent Monitoring, GenAI telemetry packages, AI agent4 evaluation telemetry, or adjacent AI infrastructure observability. Render, validate, diagnose, and5 safely apply Splunk Observability Cloud AI Agent Monitoring setup plans, including GenAI Python6 instrumentation, instrumentation-side evaluations, Log Observer Connect handoffs, histogram collector7 readiness, and AI Infrastructure Monitoring coverage."8compatibility: "Splunk Cloud Platform 10.5.2605: conditional. Follow documented package, entitlement, topology, and customer-managed runtime guardrails; self-managed paths remain on the public 10.4 baseline."9metadata:10 source: https://skillmd.com/skills/chambear2809/splunk-observability-ai-agent-monitoring-setup11 splunk_cloud_10_5: "conditional"12 compatibility_verified: "2026-08-20"13---1415# Splunk Observability AI Agent Monitoring Setup1617## Prerequisites1819| Tool or access | Purpose | Verify |20|---|---|---|21| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |22| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |23| Credential files for live modes | Keep secrets out of chat | Verify paths only |2425## Workflow Overview2627```text28┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐29│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │30└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘31```3233## When to Activate3435- Setting up or auditing Splunk AI Agent Monitoring, GenAI telemetry packages, AI agent evaluation telemetry, or36 adjacent AI infrastructure observability.37- Preview and review the splunk observability ai agent monitoring setup workflow before any live apply phase.38- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.3940## Scope4142Follow the documented read-only or render-first path whenever it is available.43This skill does not imply permission to mutate live systems. Require explicit44apply flags, protected credentials, and operator review for state changes.4546## Examples4748Inspect the supported setup modes before selecting one:4950```bash51bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/setup.sh --help52```5354Expected output: usage, supported modes, and required arguments are displayed55without changing the target environment.5657Inspect validation modes before running completion checks:5859```bash60bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/validate.sh --help61```6263Expected output: offline, live, and completion options are displayed when the64skill supports them; help exits without mutation.6566## Troubleshooting6768| Issue | Cause | Resolution |69|---|---|---|70| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |71| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |72| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |73| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |7475## Overview7677Use this skill to build a render-first setup plan for Splunk AI Agent Monitoring and adjacent AI Infrastructure Monitoring. The skill owns orchestration, validation, coverage reporting, and safe delegation; it does not mark UI-only workflows as API-applied.7879The renderer always emits a coverage report. Treat `coverage-report.json` as the source of truth for what is `delegated_apply`, `render`, `deeplink`, `handoff`, or `not_applicable`.8081## Safety Rules8283- Never ask for Splunk Observability tokens, Splunk Platform passwords, HEC tokens, LLM API keys, or client secrets in conversation.84- Never pass secrets directly on the command line or as environment-variable prefixes.85- Use file-based secrets only, such as `SPLUNK_O11Y_TOKEN_FILE`, `--o11y-token-file`, `--platform-hec-token-file`, or the delegated skill's file flags.86- Default prompt and response content capture to off.87- Require explicit accept flags before rendering content capture or LLM-as-judge evaluation env vars.88- Do not promise PII redaction, hashing, or truncation. The upstream GenAI utility marks those helpers as not implemented.8990## Primary Workflow91921. Start from `template.example`, or create a small JSON/YAML spec with the target realm, collector mode, agent frameworks, and AI infrastructure products.93 For Kubernetes runtime handoffs, set `deployment.workload_kind`, `deployment.workload_namespace`, `deployment.workload_name`, and `deployment.container_name` to the exact application workload and container that should receive the environment variables.942. Render and validate:9596 ```bash97 bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/setup.sh \98 --render \99 --validate \100 --spec skills/splunk-observability-ai-agent-monitoring-setup/template.example \101 --output-dir splunk-observability-ai-agent-monitoring-rendered102 ```1031043. Review the generated files:105 - `coverage-report.json` and `coverage-report.md`106 - `apply-plan.json`107 - `runtime/python.env` and `runtime/requirements.txt`108 - `collector/values-ai-agent-monitoring.yaml`109 - `handoff.md` and `doctor-report.md`1101114. Apply only when explicitly requested:112113 ```bash114 bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/setup.sh \115 --apply collector,hec,loc \116 --spec my-ai-agent-monitoring.json \117 --o11y-token-file /tmp/splunk_o11y_token118 ```119120## Apply Model121122Supported `--apply` sections are:123124- `collector` delegates to `splunk-observability-otel-collector-setup` using `--apply-k8s` or `--apply-linux`.125- `hec` delegates to `splunk-hec-service-setup --phase apply`.126- `loc` delegates to `splunk-observability-cloud-integration-setup --apply log_observer_connect`; the Observability UI connection wizard remains `deeplink`.127- `python-runtime` and `kubernetes-runtime` render deterministic application changes and print operator handoffs because the skill does not own the user's application repository or cluster workloads.128- `ai-infra-collector` delegates to existing product skills when available, such as Cisco AI PODs and NVIDIA GPU, otherwise renders collector overlay handoffs.129- `dashboards` delegates to `splunk-observability-dashboard-builder`.130- `detectors` delegates to `splunk-observability-native-ops`.131132## Validation Rules133134Validation must fail for:135136- `send_otlp_histograms: false`, because the Agents page requires histogram metrics.137- Prompt/response content capture without `--accept-content-capture` or `accept_content_capture: true`.138- Instrumentation-side evaluations without `--accept-evaluation-cost` or `accept_evaluation_cost: true`.139- OpenAI evaluations without `OTEL_INSTRUMENTATION_GENAI_EVALS_SEPARATE_PROCESS=true`.140- Unsupported package names such as `splunk-otel-instrumentation-openai-v2` or `splunk-otel-instrumentation-vertexai`.141- Selected packages whose Python requirement exceeds the declared runtime version.142143## References144145- Read `reference.md` for the CLI and generated artifact contract.146- Read `references/coverage.md` when changing coverage status or adding product support.147- Read `references/package-catalog.md` before changing package names, Python floors, or framework mappings.148- Read `references/apply-and-safety.md` before changing live apply behavior.149- Read `references/troubleshooting.md` when updating doctor output.