# Splunk Observability AI Agent Monitoring Setup

> Use when setting up or auditing Splunk AI Agent Monitoring, GenAI telemetry packages, AI agent evaluation telemetry, or adjacent AI infrastructure observability. Render, validate, diagnose, and safely apply Splunk Observability Cloud AI Agent Monitoring setup plans, including GenAI Python instrumentation, instrumentation-side evaluations, Log Observer Connect handoffs, histogram collector readiness, and AI Infrastructure Monitoring coverage.

- Skill: `chambear2809/splunk-observability-ai-agent-monitoring-setup` (Agent Skill, multi-file: 12 files)
- Install (CLI): `npx skillmds@latest add chambear2809/splunk-observability-ai-agent-monitoring-setup`
- Raw SKILL.md: https://api.skillmd.com/api/skills/chambear2809/splunk-observability-ai-agent-monitoring-setup/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: chambear2809 (https://skillmd.com/u/chambear2809)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/chambear2809/splunk-observability-ai-agent-monitoring-setup

---


# Splunk Observability AI Agent Monitoring Setup

## Prerequisites

| Tool or access | Purpose | Verify |
|---|---|---|
| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |
| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |
| Credential files for live modes | Keep secrets out of chat | Verify paths only |

## Workflow Overview

```text
┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘
```

## When to Activate

- Setting up or auditing Splunk AI Agent Monitoring, GenAI telemetry packages, AI agent evaluation telemetry, or
  adjacent AI infrastructure observability.
- Preview and review the splunk observability ai agent monitoring setup workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

## Scope

Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.

## Examples

Inspect the supported setup modes before selecting one:

```bash
bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/setup.sh --help
```

Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.

Inspect validation modes before running completion checks:

```bash
bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/validate.sh --help
```

Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.

## Troubleshooting

| Issue | Cause | Resolution |
|---|---|---|
| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |
| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |
| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |
| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |

## Overview

Use this skill to build a render-first setup plan for Splunk AI Agent Monitoring and adjacent AI Infrastructure Monitoring. The skill owns orchestration, validation, coverage reporting, and safe delegation; it does not mark UI-only workflows as API-applied.

The renderer always emits a coverage report. Treat `coverage-report.json` as the source of truth for what is `delegated_apply`, `render`, `deeplink`, `handoff`, or `not_applicable`.

## Safety Rules

- Never ask for Splunk Observability tokens, Splunk Platform passwords, HEC tokens, LLM API keys, or client secrets in conversation.
- Never pass secrets directly on the command line or as environment-variable prefixes.
- Use file-based secrets only, such as `SPLUNK_O11Y_TOKEN_FILE`, `--o11y-token-file`, `--platform-hec-token-file`, or the delegated skill's file flags.
- Default prompt and response content capture to off.
- Require explicit accept flags before rendering content capture or LLM-as-judge evaluation env vars.
- Do not promise PII redaction, hashing, or truncation. The upstream GenAI utility marks those helpers as not implemented.

## Primary Workflow

1. Start from `template.example`, or create a small JSON/YAML spec with the target realm, collector mode, agent frameworks, and AI infrastructure products.
   For Kubernetes runtime handoffs, set `deployment.workload_kind`, `deployment.workload_namespace`, `deployment.workload_name`, and `deployment.container_name` to the exact application workload and container that should receive the environment variables.
2. Render and validate:

   ```bash
   bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/setup.sh \
     --render \
     --validate \
     --spec skills/splunk-observability-ai-agent-monitoring-setup/template.example \
     --output-dir splunk-observability-ai-agent-monitoring-rendered
   ```

3. Review the generated files:
   - `coverage-report.json` and `coverage-report.md`
   - `apply-plan.json`
   - `runtime/python.env` and `runtime/requirements.txt`
   - `collector/values-ai-agent-monitoring.yaml`
   - `handoff.md` and `doctor-report.md`

4. Apply only when explicitly requested:

   ```bash
   bash skills/splunk-observability-ai-agent-monitoring-setup/scripts/setup.sh \
     --apply collector,hec,loc \
     --spec my-ai-agent-monitoring.json \
     --o11y-token-file /tmp/splunk_o11y_token
   ```

## Apply Model

Supported `--apply` sections are:

- `collector` delegates to `splunk-observability-otel-collector-setup` using `--apply-k8s` or `--apply-linux`.
- `hec` delegates to `splunk-hec-service-setup --phase apply`.
- `loc` delegates to `splunk-observability-cloud-integration-setup --apply log_observer_connect`; the Observability UI connection wizard remains `deeplink`.
- `python-runtime` and `kubernetes-runtime` render deterministic application changes and print operator handoffs because the skill does not own the user's application repository or cluster workloads.
- `ai-infra-collector` delegates to existing product skills when available, such as Cisco AI PODs and NVIDIA GPU, otherwise renders collector overlay handoffs.
- `dashboards` delegates to `splunk-observability-dashboard-builder`.
- `detectors` delegates to `splunk-observability-native-ops`.

## Validation Rules

Validation must fail for:

- `send_otlp_histograms: false`, because the Agents page requires histogram metrics.
- Prompt/response content capture without `--accept-content-capture` or `accept_content_capture: true`.
- Instrumentation-side evaluations without `--accept-evaluation-cost` or `accept_evaluation_cost: true`.
- OpenAI evaluations without `OTEL_INSTRUMENTATION_GENAI_EVALS_SEPARATE_PROCESS=true`.
- Unsupported package names such as `splunk-otel-instrumentation-openai-v2` or `splunk-otel-instrumentation-vertexai`.
- Selected packages whose Python requirement exceeds the declared runtime version.

## References

- Read `reference.md` for the CLI and generated artifact contract.
- Read `references/coverage.md` when changing coverage status or adding product support.
- Read `references/package-catalog.md` before changing package names, Python floors, or framework mappings.
- Read `references/apply-and-safety.md` before changing live apply behavior.
- Read `references/troubleshooting.md` when updating doctor output.