Splunk Platform <-> Splunk Observability Cloud Integration Setup
Prerequisites
| Tool or access |
Purpose |
Verify |
| Bash and Python 3 |
Run bundled setup and validation helpers |
bash --version && python3 --version |
| Required product/platform access |
Inspect or configure the selected target |
Complete the documented preflight |
| Credential files for live modes |
Keep secrets out of chat |
Verify paths only |
Workflow Overview
┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘
When to Activate
- A user asks to pair Splunk Platform with Splunk Observability Cloud, set up Unified Identity or Centralized RBAC,
configure the Discover app, install the Infrastructure Monitoring Add-on, configure Related Content or Log
Observer Connect,.
- Preview and review the splunk observability cloud integration setup workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.
Scope
Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.
Examples
Inspect the supported setup modes before selecting one:
bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh --help
Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.
Inspect validation modes before running completion checks:
bash skills/splunk-observability-cloud-integration-setup/scripts/validate.sh --help
Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.
Troubleshooting
| Issue |
Cause |
Resolution |
| Preflight fails |
A required tool or access path is missing |
Resolve it before rendering or applying |
| Rendered assets are incomplete |
Required non-secret inputs are absent |
Complete intake and render again |
| Apply is blocked |
Review, credentials, or explicit acceptance is missing |
Use the documented handoff |
| Validation is incomplete |
Live evidence is unavailable |
Record the gap and keep completion open |
TA Completion Gate
For every TA/add-on or dashboard companion run, satisfy the shared
TA completion gate: configure and enable the
data ingest path owned by this skill or its required companion, validate events
or metrics in the target indexes/source types, and verify any
pre-built/package-shipped dashboards are visible, macro-aligned, and returning
data. If the package ships no dashboards, record that evidence explicitly and
hand off dashboard use to the consuming app, ES/ITSI/ARI content, or readiness
doctor.
Single skill that pairs a Splunk Cloud Platform or Splunk Enterprise stack with
Splunk Observability Cloud and configures every navigate-into-O11y surface:
Unified Identity SSO, Centralized RBAC, the in-app Discover app, Related
Content previews in Search & Reporting, Log Observer Connect, Dashboard Studio
O11y metrics, and the Splunk Infrastructure Monitoring Add-on (sim SPL
command + streaming modular inputs).
The workflow is render-first by default. Live API changes only happen when the
user explicitly asks for --apply.
Coverage Model
Every rendered section gets an explicit coverage status:
api_apply — a documented public API supports create, update, delete, or validate.
api_validate — a documented public API supports read or validation only.
deeplink — the skill renders a deterministic Splunk / Observability UI link
and validates referenced data where an API allows.
handoff — the skill renders deterministic operator steps for UI-only or
cross-skill workflows (e.g., Splunkbase install via splunk-app-install).
install_apply — the skill installs or configures a Splunk-side companion
app via Splunkbase + REST.
not_applicable — the section does not apply to the chosen target
(e.g., UID on Splunk Enterprise, Discover app on SCP < 10.1.2507).
UI-only steps (multi-org Make Default, the SE TLS-certificate paste, the
"Override default organization" user action) render as deeplink and never
claim API parity that does not exist.
Safety Rules
- Never ask for Splunk Observability tokens, Splunk Cloud Platform admin JWTs,
Splunk passwords, SIM Add-on org tokens, or Log Observer Connect
service-account passwords in conversation.
- Never pass any secret on the command line or as an environment-variable
prefix.
- Use
--token-file for the regular Splunk Observability Cloud API token.
- Use
--admin-token-file for the Splunk Observability Cloud admin token used
by Unified Identity pairing and enable-centralized-rbac.
- Use
--org-token-file for the Splunk Observability Cloud org access token
used by the Splunk Infrastructure Monitoring Add-on account.
- Use
--service-account-password-file for the Log Observer Connect
service-account password.
- Token and password files must be regular, single-hardlink, non-empty files
with mode
600 and no more than 64 KiB. Tokens contain one printable-ASCII
value; passwords may contain printable UTF-8. Either permits at most one
trailing LF or CRLF; all other controls and Unicode line separators fail.
Authenticated
clients use O_NOFOLLOW, bounded double reads, and stable metadata/content
fingerprints; --apply aborts on any mismatch.
- Authenticated REST calls require HTTPS before credentials are attached and
refuse every redirect so Basic, bearer, JWT, and O11y token headers cannot be
forwarded to a different URL.
- Generated authenticated curl helpers put
-q first so user or system curl
configuration cannot relax TLS/redirect policy. Log Observer Connect converts
its password file directly from the validated descriptor into a private curl
config; the password is never loaded into a shell variable or reopened.
- Reject direct secret flags such as
--token, --access-token,
--api-token, --o11y-token, --admin-token, --org-token, --sf-token,
--service-account-password, and --password.
- Prefer
SPLUNK_O11Y_REALM, SPLUNK_O11Y_TOKEN_FILE,
SPLUNK_O11Y_ADMIN_TOKEN_FILE, and SPLUNK_O11Y_ORG_TOKEN_FILE from the
repo credentials file when present; these store only realms and token-file
paths, never token values.
- Strip every secret from
00-09-*.md, apply-plan.json, payloads/,
current-state.json, state/apply-state.json, and any other rendered
artifact on disk.
enable-centralized-rbac is destructive and irreversible without Splunk
Support. This repo has no safe file-backed transport for the required ACS
token, so cutover is classified as handoff and always fails before mutation.
bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_o11y_token
helps the user create a token file without exposing the secret in shell
history.
Primary Workflow
Collect non-secret values: target (cloud or enterprise), Splunk Cloud
stack, Splunk Observability Cloud realm (us0/us1/eu0/eu1/eu2/au0/jp0/sg0/
us2-gcp), multi-org list, Log Observer Connect service-account username,
indexes the LOC service account should access, Splunk Infrastructure
Monitoring Add-on account name and modular-input picks.
Create or update a JSON/YAML spec from template.example.
Render and validate:
bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \
--render \
--spec skills/splunk-observability-cloud-integration-setup/template.example \
--output-dir splunk-observability-cloud-integration-rendered
Review splunk-observability-cloud-integration-rendered/:
README.md — TL;DR and ordered next-step commands.
architecture.mmd — Mermaid topology of the rendered integration.
00-prerequisites.md through 09-handoff.md — numbered per-section plans.
coverage-report.json — per-section coverage status.
apply-plan.json — apply ordering with idempotency keys (no secrets).
payloads/ — per-step request bodies for ACS / REST calls.
scripts/ — per-step apply scripts and cross-skill handoff drivers.
support-tickets/ — pre-filled tickets when Splunk Support is required.
sim-addon/ — MTS sizing, plus the curated SignalFlow catalog files.
Apply only when explicitly requested:
bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \
--apply \
--spec skills/splunk-observability-cloud-integration-setup/template.example \
--realm us0 \
--admin-token-file /tmp/splunk_o11y_admin_token \
--org-token-file /tmp/splunk_o11y_org_token \
--service-account-password-file /tmp/loc_svc_account_password
To run only a subset of sections:
bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \
--apply pairing,sim_addon \
--spec my-integration.yaml
Centralized RBAC cutover is a fail-closed handoff because no safe file-backed
transport is implemented. The handoff never places a token on process argv.
End-User UX (the "easy to use" promise)
Five entry points, ordered by user effort:
--quickstart — renders and validates the common UID + Discover app + SIM
scenario, then prints explicit supported apply and UI/admin handoffs. It does
not mutate live state.
--render (default) — produces the numbered plan tree; never touches live
state.
--discover — writes a read-only rendered-plan inventory scaffold to
current-state.json. It does not currently claim a complete live snapshot.
--doctor — writes the static twenty-check review catalog and a prioritized
handoff/fix list. Use --validate --live for the limited implemented
token-auth and SIM-account reachability reads.
--apply SECTIONS — applies explicitly named supported sections. Live apply
without a section list is refused because plans can include UI/admin handoffs.
Plus quality-of-life flags:
--enable-token-auth — flips token authentication on if disabled (auto-
rendered as a fix from --doctor).
--explain — prints the apply plan in plain English with no API calls
(useful for change-management approvals).
--list-sim-templates / --render-sim-templates aws_ec2,kubernetes, os_hosts,apm — pick from the curated SignalFlow catalog without writing
SignalFlow.
--make-default-deeplink — emits the multi-org "Make Default" UI deeplink
for the named realm (since no API exists).
--quickstart-enterprise — renders and validates the Splunk Enterprise fast
path; supported mutations must be invoked explicitly afterward.
--rollback <section> — renders (does not auto-run) the reverse-engineered
commands for steps that have a public reversible API; for irreversible steps
(enable-centralized-rbac, deleted users) it renders a Splunk Support
ticket template instead.
Supported Sections
Specs use api_version: splunk-observability-cloud-integration-setup/v1 and
can include:
prerequisites — static region/realm and FedRAMP/GovCloud/GCP policy
rendering. Live stack version, trial-stack, operator-role, and Discover-app
10.1.2507+ checks remain explicit preflight handoffs.
token_auth — token-auth state read + flip, edit_tokens_settings
capability check.
pairing — Splunk Cloud Platform Unified Identity (UID) via POST /adminconfig/v2/observability/sso-pairing, or Discover-app API-token
connection. Multi-org is a fail-closed per-org-token handoff plus Make
Default deeplink; the skill never reuses one token across declared orgs.
Pairing is not a Splunk Enterprise section; Enterprise uses Log Observer
Connect separately.
centralized_rbac — acs observability enable-capabilities (provisions
o11y_admin / o11y_power / o11y_read_only / o11y_usage) and
enable-centralized-rbac; the o11y_access gate role; UID role mapping.
related_content_capabilities — read_o11y_content, write_o11y_content,
EXECUTE_SIGNAL_FLOW, READ_APM_DATA, READ_BASIC_UI_ACCESS, READ_EVENT
capability assignments for Real Time Metrics + previews.
discover_app — converges the non-secret Configurations tabs of the
in-platform Discover Splunk Observability Cloud app: Related Content
discovery, Field aliasing (Auto Field Mapping), and Automatic UI updates;
Test related content remains a deeplink. Access tokens are written only by
service-account pairing, preventing a duplicate connection. This section
also merges Read permission for selected roles.
log_observer_connect — service-account user + role + workload rule;
Splunk Cloud Platform path or Splunk Enterprise TLS-certificate path.
Hands off realm-IP allowlist deltas to splunk-cloud-acs-admin-setup.
dashboard_studio_o11y — default-connection + capability validations + a
starter Dashboard Studio JSON snippet using O11y metrics.
sim_addon — installs Splunk_TA_sim (Splunkbase 5247), creates the
sim_metrics index when missing, configures the SIM account through the
TA UCC custom REST handler, renders curated SignalFlow modular inputs from
the catalog (AWS_EC2, AWS_Lambda, Azure, GCP, Containers, Kubernetes,
OS_Hosts, APM_Errors, APM_Throughput, RUM, Synthetics), runs MTS sizing
preflight, and hands off the Splunk Cloud Victoria-stack search-head HEC
allowlist + the ITSI Content Pack for Splunk Observability Cloud.
enterprise_mode — collapses UID / ACS observability / Discover-app
Configurations sections to not_applicable and switches LOC to the SE
TLS-cert path.
For per-section flag references and REST payload shapes, read
reference.md and the focused docs under
references/.
Out of Scope (handed off, not duplicated)
- Splunk Add-on for OpenTelemetry Collector (Splunkbase 7125) — handled by
splunk-observability-otel-collector-setup.
- Splunk Synthetic Monitoring Add-on (Splunkbase
5608) — archived and not
listed for Splunk 10.5. Do not install it on a new or upgraded 10.5 stack.
Use SIM Add-on streams for in-platform metrics and native Splunk
Observability Cloud Synthetics workflows for browser/API tests.
- Splunk On-Call wiring — handled by
splunk-oncall-setup.
- ITSI Content Pack content management — handled by
splunk-itsi-config.
- Splunk Observability Cloud dashboards / detectors / Synthetics / RUM CRUD —
handled by
splunk-observability-dashboard-builder and
splunk-observability-native-ops.
- AppDynamics for Log Observer Connect — separate AppDynamics SaaS workflow.
Scenarios Gallery
Six worked end-to-end examples, copy/paste-ready:
- Cloud quickstart (greenfield) —
--quickstart renders and validates a
fresh SCP plan, then prints supported apply and Related Content handoffs.
- Multi-org Cloud — renders a fail-closed, distinct-token-per-org handoff
for three O11y orgs on one SCP stack; default-org selection uses a deeplink.
- Cloud API-token mode (no UID) — user API access-token pairing (no
admin token required); SIM Add-on plus Related Content handoff; appropriate
for stacks where UID is not yet in scope.
- Migrate API-token -> UID — an existing API-token customer wants Unified Identity;
renderer detects the existing connection and renders a numbered migration
plan (pair UID, validate, instruct user to delete old SA via Discover
app deeplink, optionally
enable-centralized-rbac).
- Splunk Enterprise —
--quickstart-enterprise renders SIM plus LOC
service-account/TLS assets without mutating; UID/RBAC/Discover-app sections are
marked not_applicable.
- Inherit existing integration — use
--discover for a rendered-plan
inventory scaffold, --validate --live for limited reachability, then --doctor
to identify drift and gaps, then targeted --apply <section> to
converge to the rendered plan.
Useful Commands
Validate a draft spec:
bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \
--validate \
--spec skills/splunk-observability-cloud-integration-setup/template.example
Render without applying:
bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \
--render \
--spec skills/splunk-observability-cloud-integration-setup/template.example \
--output-dir splunk-observability-cloud-integration-rendered
Diagnose an existing integration:
bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \
--doctor \
--realm us0 \
--admin-token-file /tmp/splunk_o11y_admin_token
List the curated SignalFlow modular-input catalog:
bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \
--list-sim-templates
Hand-offs to Other Skills
- App install ->
skills/splunk-app-install/scripts/install_app.sh --source splunkbase --app-id 5247 (Splunk_TA_sim).
- ACS Log Observer Connect realm-IP allowlist deltas ->
skills/splunk-cloud-acs-admin-setup/scripts/setup.sh --phase render --target-stack STACK_NAME --features search-api --search-api-subnets <pre-baked-realm-IPs>.
- ACS Splunk Cloud Victoria-stack search-head HEC allowlist (SIM Add-on
prerequisite) ->
skills/splunk-cloud-acs-admin-setup/scripts/setup.sh --phase render --target-stack STACK_NAME --features hec.
- ITSI Content Pack for Splunk Observability Cloud ->
skills/splunk-itsi-config/SKILL.md.
- Splunk Observability Cloud dashboards, detectors, Log Observer Connect
queries, Synthetics, RUM ->
skills/splunk-observability-dashboard-builder/SKILL.md and
skills/splunk-observability-native-ops/SKILL.md.
- OTel collection on Kubernetes and Linux ->
skills/splunk-observability-otel-collector-setup/SKILL.md.
- Splunk On-Call detector recipients ->
skills/splunk-oncall-setup/SKILL.md.
Compliance and Security Baseline
- Splunk Cloud Platform Unified Identity is supported in AWS regions only;
GovCloud and GCP regions are excluded. The skill marks UID sections
not_applicable when GovCloud or GCP is detected and renders a Service
Account fallback plan.
- Cross-region pairing (e.g., us0 realm to us-west-2 region) requires Splunk
Account team approval; the preflight WARNs and emits a
support-tickets/cross-region-pairing.md template.
- FedRAMP / IL5 customers cannot use UID against the public commercial O11y
realms; the skill renders a
support-tickets/fedramp-il5-readiness.md
template instead of attempting the pair call.
- The skill never asks for nor logs secret material, refuses every direct
secret CLI flag, and redacts every token, password, JWT, and authorization
value from rendered artifacts. Non-secret pairing job IDs are retained in
mode-600 apply state so asynchronous status polling can resume safely.
MCP Tools
This skill includes checked-in, read-only Splunk MCP custom tools generated
from mcp_tools.source.yaml.
Validate or regenerate the tool artifact:
python3 skills/shared/scripts/mcp_tools.py validate skills/splunk-observability-cloud-integration-setup
python3 skills/shared/scripts/mcp_tools.py generate skills/splunk-observability-cloud-integration-setup
Load the tools into Splunk MCP Server:
bash skills/splunk-observability-cloud-integration-setup/scripts/load_mcp_tools.sh
The loader uses the supported /mcp_tools REST batch endpoint by default. Use
--allow-legacy-kv only for older MCP Server app versions that lack that
endpoint.
1---2name: splunk-observability-cloud-integration-setup3description: "Use when a user asks to pair Splunk Platform with Splunk Observability Cloud, set up Unified Identity or4 Centralized RBAC, configure the Discover app, install the Infrastructure Monitoring Add-on, configure5 Related Content or Log Observer Connect, bring O11y metrics into Splunk with sim, or navigate from6 Splunk Platform into Observability workflows. Render, preflight, apply, validate, and diagnose Splunk7 Platform to Splunk Observability Cloud pairing for Splunk Cloud Platform and Splunk Enterprise. Covers8 token-auth enablement, realm checks, Unified Identity or service-account pairing, multi-org defaults,9 Centralized RBAC, Discover Splunk Observability Cloud app configuration, Log Observer Connect, Related10 Content, Real Time Metrics, Dashboard Studio O11y metrics, and Splunk_TA_sim modular inputs."11compatibility: "Splunk Cloud Platform 10.5.2605: conditional. Follow documented package, entitlement, topology, and customer-managed runtime guardrails; self-managed paths remain on the public 10.4 baseline."12metadata:13 source: https://skillmd.com/skills/chambear2809/splunk-observability-cloud-integration-setup14 splunk_cloud_10_5: "conditional"15 compatibility_verified: "2026-08-20"16---1718# Splunk Platform <-> Splunk Observability Cloud Integration Setup1920## Prerequisites2122| Tool or access | Purpose | Verify |23|---|---|---|24| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |25| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |26| Credential files for live modes | Keep secrets out of chat | Verify paths only |2728## Workflow Overview2930```text31┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐32│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │33└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘34```3536## When to Activate3738- A user asks to pair Splunk Platform with Splunk Observability Cloud, set up Unified Identity or Centralized RBAC,39 configure the Discover app, install the Infrastructure Monitoring Add-on, configure Related Content or Log40 Observer Connect,.41- Preview and review the splunk observability cloud integration setup workflow before any live apply phase.42- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.4344## Scope4546Follow the documented read-only or render-first path whenever it is available.47This skill does not imply permission to mutate live systems. Require explicit48apply flags, protected credentials, and operator review for state changes.4950## Examples5152Inspect the supported setup modes before selecting one:5354```bash55bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh --help56```5758Expected output: usage, supported modes, and required arguments are displayed59without changing the target environment.6061Inspect validation modes before running completion checks:6263```bash64bash skills/splunk-observability-cloud-integration-setup/scripts/validate.sh --help65```6667Expected output: offline, live, and completion options are displayed when the68skill supports them; help exits without mutation.6970## Troubleshooting7172| Issue | Cause | Resolution |73|---|---|---|74| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |75| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |76| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |77| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |7879## TA Completion Gate8081For every TA/add-on or dashboard companion run, satisfy the shared82[TA completion gate](../shared/ta_completion_gate.md): configure and enable the83data ingest path owned by this skill or its required companion, validate events84or metrics in the target indexes/source types, and verify any85pre-built/package-shipped dashboards are visible, macro-aligned, and returning86data. If the package ships no dashboards, record that evidence explicitly and87hand off dashboard use to the consuming app, ES/ITSI/ARI content, or readiness88doctor.8990Single skill that pairs a Splunk Cloud Platform or Splunk Enterprise stack with91Splunk Observability Cloud and configures every navigate-into-O11y surface:92Unified Identity SSO, Centralized RBAC, the in-app Discover app, Related93Content previews in Search & Reporting, Log Observer Connect, Dashboard Studio94O11y metrics, and the Splunk Infrastructure Monitoring Add-on (`sim` SPL95command + streaming modular inputs).9697The workflow is render-first by default. Live API changes only happen when the98user explicitly asks for `--apply`.99100## Coverage Model101102Every rendered section gets an explicit coverage status:103104- `api_apply` — a documented public API supports create, update, delete, or validate.105- `api_validate` — a documented public API supports read or validation only.106- `deeplink` — the skill renders a deterministic Splunk / Observability UI link107 and validates referenced data where an API allows.108- `handoff` — the skill renders deterministic operator steps for UI-only or109 cross-skill workflows (e.g., Splunkbase install via `splunk-app-install`).110- `install_apply` — the skill installs or configures a Splunk-side companion111 app via Splunkbase + REST.112- `not_applicable` — the section does not apply to the chosen target113 (e.g., UID on Splunk Enterprise, Discover app on SCP < 10.1.2507).114115UI-only steps (multi-org `Make Default`, the SE TLS-certificate paste, the116"Override default organization" user action) render as `deeplink` and never117claim API parity that does not exist.118119## Safety Rules120121- Never ask for Splunk Observability tokens, Splunk Cloud Platform admin JWTs,122 Splunk passwords, SIM Add-on org tokens, or Log Observer Connect123 service-account passwords in conversation.124- Never pass any secret on the command line or as an environment-variable125 prefix.126- Use `--token-file` for the regular Splunk Observability Cloud API token.127- Use `--admin-token-file` for the Splunk Observability Cloud admin token used128 by Unified Identity pairing and `enable-centralized-rbac`.129- Use `--org-token-file` for the Splunk Observability Cloud org access token130 used by the Splunk Infrastructure Monitoring Add-on account.131- Use `--service-account-password-file` for the Log Observer Connect132 service-account password.133- Token and password files must be regular, single-hardlink, non-empty files134 with mode `600` and no more than 64 KiB. Tokens contain one printable-ASCII135 value; passwords may contain printable UTF-8. Either permits at most one136 trailing LF or CRLF; all other controls and Unicode line separators fail.137 Authenticated138 clients use `O_NOFOLLOW`, bounded double reads, and stable metadata/content139 fingerprints; `--apply` aborts on any mismatch.140- Authenticated REST calls require HTTPS before credentials are attached and141 refuse every redirect so Basic, bearer, JWT, and O11y token headers cannot be142 forwarded to a different URL.143- Generated authenticated curl helpers put `-q` first so user or system curl144 configuration cannot relax TLS/redirect policy. Log Observer Connect converts145 its password file directly from the validated descriptor into a private curl146 config; the password is never loaded into a shell variable or reopened.147- Reject direct secret flags such as `--token`, `--access-token`,148 `--api-token`, `--o11y-token`, `--admin-token`, `--org-token`, `--sf-token`,149 `--service-account-password`, and `--password`.150- Prefer `SPLUNK_O11Y_REALM`, `SPLUNK_O11Y_TOKEN_FILE`,151 `SPLUNK_O11Y_ADMIN_TOKEN_FILE`, and `SPLUNK_O11Y_ORG_TOKEN_FILE` from the152 repo `credentials` file when present; these store only realms and token-file153 paths, never token values.154- Strip every secret from `00-09-*.md`, `apply-plan.json`, `payloads/`,155 `current-state.json`, `state/apply-state.json`, and any other rendered156 artifact on disk.157- `enable-centralized-rbac` is destructive and irreversible without Splunk158 Support. This repo has no safe file-backed transport for the required ACS159 token, so cutover is classified as `handoff` and always fails before mutation.160- `bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_o11y_token`161 helps the user create a token file without exposing the secret in shell162 history.163164## Primary Workflow1651661. Collect non-secret values: target (cloud or enterprise), Splunk Cloud167 stack, Splunk Observability Cloud realm (us0/us1/eu0/eu1/eu2/au0/jp0/sg0/168 us2-gcp), multi-org list, Log Observer Connect service-account username,169 indexes the LOC service account should access, Splunk Infrastructure170 Monitoring Add-on account name and modular-input picks.1712. Create or update a JSON/YAML spec from `template.example`.1723. Render and validate:173174 ```bash175 bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \176 --render \177 --spec skills/splunk-observability-cloud-integration-setup/template.example \178 --output-dir splunk-observability-cloud-integration-rendered179 ```1801814. Review `splunk-observability-cloud-integration-rendered/`:182 - `README.md` — TL;DR and ordered next-step commands.183 - `architecture.mmd` — Mermaid topology of the rendered integration.184 - `00-prerequisites.md` through `09-handoff.md` — numbered per-section plans.185 - `coverage-report.json` — per-section coverage status.186 - `apply-plan.json` — apply ordering with idempotency keys (no secrets).187 - `payloads/` — per-step request bodies for ACS / REST calls.188 - `scripts/` — per-step apply scripts and cross-skill handoff drivers.189 - `support-tickets/` — pre-filled tickets when Splunk Support is required.190 - `sim-addon/` — MTS sizing, plus the curated SignalFlow catalog files.1911925. Apply only when explicitly requested:193194 ```bash195 bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \196 --apply \197 --spec skills/splunk-observability-cloud-integration-setup/template.example \198 --realm us0 \199 --admin-token-file /tmp/splunk_o11y_admin_token \200 --org-token-file /tmp/splunk_o11y_org_token \201 --service-account-password-file /tmp/loc_svc_account_password202 ```203204 To run only a subset of sections:205206 ```bash207 bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \208 --apply pairing,sim_addon \209 --spec my-integration.yaml210 ```211212 Centralized RBAC cutover is a fail-closed handoff because no safe file-backed213 transport is implemented. The handoff never places a token on process argv.214215## End-User UX (the "easy to use" promise)216217Five entry points, ordered by user effort:218219- `--quickstart` — renders and validates the common UID + Discover app + SIM220 scenario, then prints explicit supported apply and UI/admin handoffs. It does221 not mutate live state.222- `--render` (default) — produces the numbered plan tree; never touches live223 state.224- `--discover` — writes a read-only rendered-plan inventory scaffold to225 `current-state.json`. It does not currently claim a complete live snapshot.226- `--doctor` — writes the static twenty-check review catalog and a prioritized227 handoff/fix list. Use `--validate --live` for the limited implemented228 token-auth and SIM-account reachability reads.229- `--apply SECTIONS` — applies explicitly named supported sections. Live apply230 without a section list is refused because plans can include UI/admin handoffs.231232Plus quality-of-life flags:233234- `--enable-token-auth` — flips token authentication on if disabled (auto-235 rendered as a fix from `--doctor`).236- `--explain` — prints the apply plan in plain English with no API calls237 (useful for change-management approvals).238- `--list-sim-templates` / `--render-sim-templates aws_ec2,kubernetes,239 os_hosts,apm` — pick from the curated SignalFlow catalog without writing240 SignalFlow.241- `--make-default-deeplink` — emits the multi-org "Make Default" UI deeplink242 for the named realm (since no API exists).243- `--quickstart-enterprise` — renders and validates the Splunk Enterprise fast244 path; supported mutations must be invoked explicitly afterward.245- `--rollback <section>` — renders (does not auto-run) the reverse-engineered246 commands for steps that have a public reversible API; for irreversible steps247 (`enable-centralized-rbac`, deleted users) it renders a Splunk Support248 ticket template instead.249250## Supported Sections251252Specs use `api_version: splunk-observability-cloud-integration-setup/v1` and253can include:254255- `prerequisites` — static region/realm and FedRAMP/GovCloud/GCP policy256 rendering. Live stack version, trial-stack, operator-role, and Discover-app257 10.1.2507+ checks remain explicit preflight handoffs.258- `token_auth` — token-auth state read + flip, `edit_tokens_settings`259 capability check.260- `pairing` — Splunk Cloud Platform Unified Identity (UID) via `POST261 /adminconfig/v2/observability/sso-pairing`, or Discover-app API-token262 connection. Multi-org is a fail-closed per-org-token handoff plus Make263 Default deeplink; the skill never reuses one token across declared orgs.264 Pairing is not a Splunk Enterprise section; Enterprise uses Log Observer265 Connect separately.266- `centralized_rbac` — `acs observability enable-capabilities` (provisions267 `o11y_admin / o11y_power / o11y_read_only / o11y_usage`) and268 `enable-centralized-rbac`; the `o11y_access` gate role; UID role mapping.269- `related_content_capabilities` — `read_o11y_content`, `write_o11y_content`,270 `EXECUTE_SIGNAL_FLOW`, `READ_APM_DATA`, `READ_BASIC_UI_ACCESS`, `READ_EVENT`271 capability assignments for Real Time Metrics + previews.272- `discover_app` — converges the non-secret Configurations tabs of the273 in-platform Discover Splunk Observability Cloud app: Related Content274 discovery, Field aliasing (Auto Field Mapping), and Automatic UI updates;275 Test related content remains a deeplink. Access tokens are written only by276 service-account `pairing`, preventing a duplicate connection. This section277 also merges Read permission for selected roles.278- `log_observer_connect` — service-account user + role + workload rule;279 Splunk Cloud Platform path or Splunk Enterprise TLS-certificate path.280 Hands off realm-IP allowlist deltas to `splunk-cloud-acs-admin-setup`.281- `dashboard_studio_o11y` — default-connection + capability validations + a282 starter Dashboard Studio JSON snippet using O11y metrics.283- `sim_addon` — installs `Splunk_TA_sim` (Splunkbase 5247), creates the284 `sim_metrics` index when missing, configures the SIM account through the285 TA UCC custom REST handler, renders curated SignalFlow modular inputs from286 the catalog (AWS_EC2, AWS_Lambda, Azure, GCP, Containers, Kubernetes,287 OS_Hosts, APM_Errors, APM_Throughput, RUM, Synthetics), runs MTS sizing288 preflight, and hands off the Splunk Cloud Victoria-stack search-head HEC289 allowlist + the ITSI Content Pack for Splunk Observability Cloud.290- `enterprise_mode` — collapses UID / ACS observability / Discover-app291 Configurations sections to `not_applicable` and switches LOC to the SE292 TLS-cert path.293294For per-section flag references and REST payload shapes, read295[reference.md](reference.md) and the focused docs under296[references/](references/).297298## Out of Scope (handed off, not duplicated)299300- Splunk Add-on for OpenTelemetry Collector (Splunkbase 7125) — handled by301 `splunk-observability-otel-collector-setup`.302- Splunk Synthetic Monitoring Add-on (Splunkbase `5608`) — archived and not303 listed for Splunk 10.5. Do not install it on a new or upgraded 10.5 stack.304 Use SIM Add-on streams for in-platform metrics and native Splunk305 Observability Cloud Synthetics workflows for browser/API tests.306- Splunk On-Call wiring — handled by `splunk-oncall-setup`.307- ITSI Content Pack content management — handled by `splunk-itsi-config`.308- Splunk Observability Cloud dashboards / detectors / Synthetics / RUM CRUD —309 handled by `splunk-observability-dashboard-builder` and310 `splunk-observability-native-ops`.311- AppDynamics for Log Observer Connect — separate AppDynamics SaaS workflow.312313## Scenarios Gallery314315Six worked end-to-end examples, copy/paste-ready:3163171. **Cloud quickstart (greenfield)** — `--quickstart` renders and validates a318 fresh SCP plan, then prints supported apply and Related Content handoffs.3192. **Multi-org Cloud** — renders a fail-closed, distinct-token-per-org handoff320 for three O11y orgs on one SCP stack; default-org selection uses a deeplink.3213. **Cloud API-token mode (no UID)** — user API access-token pairing (no322 admin token required); SIM Add-on plus Related Content handoff; appropriate323 for stacks where UID is not yet in scope.3244. **Migrate API-token -> UID** — an existing API-token customer wants Unified Identity;325 renderer detects the existing connection and renders a numbered migration326 plan (pair UID, validate, instruct user to delete old SA via Discover327 app deeplink, optionally `enable-centralized-rbac`).3285. **Splunk Enterprise** — `--quickstart-enterprise` renders SIM plus LOC329 service-account/TLS assets without mutating; UID/RBAC/Discover-app sections are330 marked `not_applicable`.3316. **Inherit existing integration** — use `--discover` for a rendered-plan332 inventory scaffold, `--validate --live` for limited reachability, then `--doctor`333 to identify drift and gaps, then targeted `--apply <section>` to334 converge to the rendered plan.335336## Useful Commands337338Validate a draft spec:339340```bash341bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \342 --validate \343 --spec skills/splunk-observability-cloud-integration-setup/template.example344```345346Render without applying:347348```bash349bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \350 --render \351 --spec skills/splunk-observability-cloud-integration-setup/template.example \352 --output-dir splunk-observability-cloud-integration-rendered353```354355Diagnose an existing integration:356357```bash358bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \359 --doctor \360 --realm us0 \361 --admin-token-file /tmp/splunk_o11y_admin_token362```363364List the curated SignalFlow modular-input catalog:365366```bash367bash skills/splunk-observability-cloud-integration-setup/scripts/setup.sh \368 --list-sim-templates369```370371## Hand-offs to Other Skills372373- App install -> `skills/splunk-app-install/scripts/install_app.sh --source374 splunkbase --app-id 5247` (Splunk_TA_sim).375- ACS Log Observer Connect realm-IP allowlist deltas ->376 `skills/splunk-cloud-acs-admin-setup/scripts/setup.sh --phase render377 --target-stack STACK_NAME --features search-api --search-api-subnets378 <pre-baked-realm-IPs>`.379- ACS Splunk Cloud Victoria-stack search-head HEC allowlist (SIM Add-on380 prerequisite) -> `skills/splunk-cloud-acs-admin-setup/scripts/setup.sh381 --phase render --target-stack STACK_NAME --features hec`.382- ITSI Content Pack for Splunk Observability Cloud ->383 `skills/splunk-itsi-config/SKILL.md`.384- Splunk Observability Cloud dashboards, detectors, Log Observer Connect385 queries, Synthetics, RUM ->386 `skills/splunk-observability-dashboard-builder/SKILL.md` and387 `skills/splunk-observability-native-ops/SKILL.md`.388- OTel collection on Kubernetes and Linux ->389 `skills/splunk-observability-otel-collector-setup/SKILL.md`.390- Splunk On-Call detector recipients -> `skills/splunk-oncall-setup/SKILL.md`.391392## Compliance and Security Baseline393394- Splunk Cloud Platform Unified Identity is supported in AWS regions only;395 GovCloud and GCP regions are excluded. The skill marks UID sections396 `not_applicable` when GovCloud or GCP is detected and renders a Service397 Account fallback plan.398- Cross-region pairing (e.g., us0 realm to us-west-2 region) requires Splunk399 Account team approval; the preflight WARNs and emits a400 `support-tickets/cross-region-pairing.md` template.401- FedRAMP / IL5 customers cannot use UID against the public commercial O11y402 realms; the skill renders a `support-tickets/fedramp-il5-readiness.md`403 template instead of attempting the pair call.404- The skill never asks for nor logs secret material, refuses every direct405 secret CLI flag, and redacts every token, password, JWT, and authorization406 value from rendered artifacts. Non-secret pairing job IDs are retained in407 mode-600 apply state so asynchronous status polling can resume safely.408409## MCP Tools410411This skill includes checked-in, read-only Splunk MCP custom tools generated412from `mcp_tools.source.yaml`.413414Validate or regenerate the tool artifact:415416```bash417python3 skills/shared/scripts/mcp_tools.py validate skills/splunk-observability-cloud-integration-setup418python3 skills/shared/scripts/mcp_tools.py generate skills/splunk-observability-cloud-integration-setup419```420421Load the tools into Splunk MCP Server:422423```bash424bash skills/splunk-observability-cloud-integration-setup/scripts/load_mcp_tools.sh425```426427The loader uses the supported `/mcp_tools` REST batch endpoint by default. Use428`--allow-legacy-kv` only for older MCP Server app versions that lack that429endpoint.