Splunk Observability Native Ops
Prerequisites
| Tool or access |
Purpose |
Verify |
| Bash and Python 3 |
Run bundled setup and validation helpers |
bash --version && python3 --version |
| Required product/platform access |
Inspect or configure the selected target |
Complete the documented preflight |
| Credential files for live modes |
Keep secrets out of chat |
Verify paths only |
Workflow Overview
┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘
When to Activate
- Configuring native Splunk Observability Cloud operations beyond collection and classic dashboards, including
detectors, alert routing, On-Call handoffs, APM service maps and traces, RUM session workflows, Synthetic tests
and waterfall.
- Preview and review the splunk observability native ops workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.
Scope
Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.
Examples
Inspect the supported setup modes before selecting one:
bash skills/splunk-observability-native-ops/scripts/setup.sh --help
Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.
Inspect validation modes before running completion checks:
bash skills/splunk-observability-native-ops/scripts/validate.sh --help
Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.
Troubleshooting
| Issue |
Cause |
Resolution |
| Preflight fails |
A required tool or access path is missing |
Resolve it before rendering or applying |
| Rendered assets are incomplete |
Required non-secret inputs are absent |
Complete intake and render again |
| Apply is blocked |
Review, credentials, or explicit acceptance is missing |
Use the documented handoff |
| Validation is incomplete |
Live evidence is unavailable |
Record the gap and keep completion open |
Use this skill for native Splunk Observability Cloud operations that are not
covered by the OTel Collector deployment skill or the classic dashboard builder.
The workflow is render-first by default. Live API changes only happen when the
user explicitly asks for --apply.
Coverage Model
Every rendered object gets an explicit coverage status:
api_apply: a documented public API supports create, update, delete, or validate.
api_validate: a documented public API supports read or validation only.
deeplink: the skill renders a deterministic Observability UI link and validates
referenced data where an API allows.
handoff: the skill renders deterministic operator steps for UI-only or app-side
workflows.
Do not mark UI-only workflows as api_apply.
Safety Rules
- Never ask for Splunk Observability tokens, passwords, or client secrets in
conversation.
- Never pass tokens or API keys on the command line or as environment-variable
prefixes.
- Use
--token-file for Splunk Observability API tokens.
- Prefer
SPLUNK_O11Y_REALM and SPLUNK_O11Y_TOKEN_FILE from the repo
credentials file when present.
- Reject direct secret flags such as
--token, --access-token, --api-token,
and --sf-token.
Splunk On-Call API actions, the REST endpoint integration, the generic email
endpoint, and the Splunk-side companion apps (Splunkbase 3546, 4886, 5863) all
live in the dedicated splunk-oncall-setup skill. This skill renders only a
deeplink-only handoff for the on_call section.
Primary Workflow
Collect non-secret values: realm, detector names, team names, service names,
environment filters, test IDs, trace IDs, Log Observer query text, and On-Call
schedule names.
Create or update a JSON/YAML spec using
templates/native-ops.example.yaml as the starting point.
Render and validate:
bash skills/splunk-observability-native-ops/scripts/setup.sh \
--render \
--validate \
--spec skills/splunk-observability-native-ops/templates/native-ops.example.yaml \
--output-dir splunk-observability-native-rendered \
--realm us0
Review coverage-report.json, apply-plan.json, deeplinks.json, and
handoff.md.
Apply only when explicitly requested. Point --spec at the same file you
rendered and validated (an absolute path, a repo-relative path like
skills/splunk-observability-native-ops/templates/native-ops.example.yaml,
or a working copy you maintain alongside your credentials):
bash skills/splunk-observability-native-ops/scripts/setup.sh \
--apply \
--spec skills/splunk-observability-native-ops/templates/native-ops.example.yaml \
--realm us0 \
--token-file /tmp/splunk_o11y_token
Supported Sections
Specs use api_version: splunk-observability-native-ops/v1 and can include:
teams: Observability teams and notification policy payloads.
detectors: SignalFlow detector definitions, rules, notifications, teams,
runbook URLs, tags, and min/max delay.
alert_routing: integrations, detector-recipient updates, team notification
policy updates, and handoffs for external routing.
muting_rules: alert muting rules.
slo_links: SLO payloads or links to existing SLOs.
synthetics: Synthetic browser, API, HTTP, SSL, and port tests; locations;
downtime windows; variables; run-now requests; run/artifact retrieval plans.
apm: service topology, service-map deeplinks, trace download plans, Trace
Analyzer deeplinks, and business workflow handoffs.
rum: session search deeplinks, replay setup handoffs, and RUM/APM links.
logs: modern logs-chart intent handoffs using SPL1, SPL2, or JSON query specs.
on_call: deeplink-only handoff that points operators to the dedicated
splunk-oncall-setup skill. Use that skill for teams, users, rotations,
escalation policies, routing keys, paging policies, alert rules, maintenance
mode, incidents, REST endpoint alerts, and Splunk-side companion app
installs (Splunkbase 3546, 4886, 5863).
For API endpoint details and current support boundaries, read
references/coverage.md when the request touches a new or ambiguous surface.
1---2name: splunk-observability-native-ops3description: "Use when configuring native Splunk Observability Cloud operations beyond collection and classic4 dashboards, including detectors, alert routing, On-Call handoffs, APM service maps and traces, RUM5 session workflows, Synthetic tests and waterfall artifacts, and modern logs chart handoffs."6compatibility: "No direct Splunk Platform runtime dependency. This workflow can be used alongside Splunk Cloud Platform 10.5.2605 through its documented external APIs or handoffs."7metadata:8 source: https://skillmd.com/skills/chambear2809/splunk-observability-native-ops9 splunk_cloud_10_5: "not-applicable"10 compatibility_verified: "2026-08-20"11---1213# Splunk Observability Native Ops1415## Prerequisites1617| Tool or access | Purpose | Verify |18|---|---|---|19| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |20| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |21| Credential files for live modes | Keep secrets out of chat | Verify paths only |2223## Workflow Overview2425```text26┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐27│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │28└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘29```3031## When to Activate3233- Configuring native Splunk Observability Cloud operations beyond collection and classic dashboards, including34 detectors, alert routing, On-Call handoffs, APM service maps and traces, RUM session workflows, Synthetic tests35 and waterfall.36- Preview and review the splunk observability native ops workflow before any live apply phase.37- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.3839## Scope4041Follow the documented read-only or render-first path whenever it is available.42This skill does not imply permission to mutate live systems. Require explicit43apply flags, protected credentials, and operator review for state changes.4445## Examples4647Inspect the supported setup modes before selecting one:4849```bash50bash skills/splunk-observability-native-ops/scripts/setup.sh --help51```5253Expected output: usage, supported modes, and required arguments are displayed54without changing the target environment.5556Inspect validation modes before running completion checks:5758```bash59bash skills/splunk-observability-native-ops/scripts/validate.sh --help60```6162Expected output: offline, live, and completion options are displayed when the63skill supports them; help exits without mutation.6465## Troubleshooting6667| Issue | Cause | Resolution |68|---|---|---|69| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |70| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |71| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |72| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |7374Use this skill for native Splunk Observability Cloud operations that are not75covered by the OTel Collector deployment skill or the classic dashboard builder.7677The workflow is render-first by default. Live API changes only happen when the78user explicitly asks for `--apply`.7980## Coverage Model8182Every rendered object gets an explicit coverage status:8384- `api_apply`: a documented public API supports create, update, delete, or validate.85- `api_validate`: a documented public API supports read or validation only.86- `deeplink`: the skill renders a deterministic Observability UI link and validates87 referenced data where an API allows.88- `handoff`: the skill renders deterministic operator steps for UI-only or app-side89 workflows.9091Do not mark UI-only workflows as `api_apply`.9293## Safety Rules9495- Never ask for Splunk Observability tokens, passwords, or client secrets in96 conversation.97- Never pass tokens or API keys on the command line or as environment-variable98 prefixes.99- Use `--token-file` for Splunk Observability API tokens.100- Prefer `SPLUNK_O11Y_REALM` and `SPLUNK_O11Y_TOKEN_FILE` from the repo101 `credentials` file when present.102- Reject direct secret flags such as `--token`, `--access-token`, `--api-token`,103 and `--sf-token`.104105Splunk On-Call API actions, the REST endpoint integration, the generic email106endpoint, and the Splunk-side companion apps (Splunkbase 3546, 4886, 5863) all107live in the dedicated `splunk-oncall-setup` skill. This skill renders only a108deeplink-only handoff for the `on_call` section.109110## Primary Workflow1111121. Collect non-secret values: realm, detector names, team names, service names,113 environment filters, test IDs, trace IDs, Log Observer query text, and On-Call114 schedule names.1152. Create or update a JSON/YAML spec using116 `templates/native-ops.example.yaml` as the starting point.1173. Render and validate:118119 ```bash120 bash skills/splunk-observability-native-ops/scripts/setup.sh \121 --render \122 --validate \123 --spec skills/splunk-observability-native-ops/templates/native-ops.example.yaml \124 --output-dir splunk-observability-native-rendered \125 --realm us0126 ```1271284. Review `coverage-report.json`, `apply-plan.json`, `deeplinks.json`, and129 `handoff.md`.1305. Apply only when explicitly requested. Point `--spec` at the same file you131 rendered and validated (an absolute path, a repo-relative path like132 `skills/splunk-observability-native-ops/templates/native-ops.example.yaml`,133 or a working copy you maintain alongside your credentials):134135 ```bash136 bash skills/splunk-observability-native-ops/scripts/setup.sh \137 --apply \138 --spec skills/splunk-observability-native-ops/templates/native-ops.example.yaml \139 --realm us0 \140 --token-file /tmp/splunk_o11y_token141 ```142143## Supported Sections144145Specs use `api_version: splunk-observability-native-ops/v1` and can include:146147- `teams`: Observability teams and notification policy payloads.148- `detectors`: SignalFlow detector definitions, rules, notifications, teams,149 runbook URLs, tags, and min/max delay.150- `alert_routing`: integrations, detector-recipient updates, team notification151 policy updates, and handoffs for external routing.152- `muting_rules`: alert muting rules.153- `slo_links`: SLO payloads or links to existing SLOs.154- `synthetics`: Synthetic browser, API, HTTP, SSL, and port tests; locations;155 downtime windows; variables; run-now requests; run/artifact retrieval plans.156- `apm`: service topology, service-map deeplinks, trace download plans, Trace157 Analyzer deeplinks, and business workflow handoffs.158- `rum`: session search deeplinks, replay setup handoffs, and RUM/APM links.159- `logs`: modern logs-chart intent handoffs using SPL1, SPL2, or JSON query specs.160- `on_call`: deeplink-only handoff that points operators to the dedicated161 `splunk-oncall-setup` skill. Use that skill for teams, users, rotations,162 escalation policies, routing keys, paging policies, alert rules, maintenance163 mode, incidents, REST endpoint alerts, and Splunk-side companion app164 installs (Splunkbase 3546, 4886, 5863).165166For API endpoint details and current support boundaries, read167`references/coverage.md` when the request touches a new or ambiguous surface.