Splunk Soar Setup

Splunk SOAR Setup

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Install Splunk SOAR On-prem, build a SOAR cluster, onboard SOAR Cloud, install Automation Broker, install splunk- side SOAR apps, or wire up SOAR with Splunk Enterprise Security.
• Preview and review the splunk soar setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-soar-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-soar-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Shared add-on completion gate

Whenever this workflow installs, configures, or hands off a Splunk-side SOAR app or add-on, follow the shared completion gate. Package delivery alone is not success; validate applicable event/action flow and shipped views, or record explicit package evidence that no dashboards ship.

This skill covers every documented Splunk SOAR install path:

• On-prem single instance (unprivileged) — soar-prepare-system followed by soar-install.
• On-prem cluster (>= 3 nodes) — make_cluster_node.pyc plus external PostgreSQL / GlusterFS / Elasticsearch / HAProxy.
• SOAR Cloud — Splunk-provisioned tenant; the skill handles JWT capture, IP allowlist, REST automation-user / token creation, and Automation Broker install for connecting back to a private network.
• Splunk-side integration — installs Splunk App for SOAR (Splunkbase
  6361. and Splunk App for SOAR Export (Splunkbase 3411) via the existing skills/splunk-app-install skill.
• ES integration readiness — calls skills/splunk-enterprise-security-config integrations.soar engine path.

Agent Behavior — Credentials

Never paste passwords, JWT tokens, or pgbouncer/postgres passwords into chat.

bash skills/shared/scripts/write_secret_file.sh /tmp/soar_admin_password
bash skills/shared/scripts/write_secret_file.sh /tmp/soar_api_token
bash skills/shared/scripts/write_secret_file.sh /tmp/postgres_master_password
bash skills/shared/scripts/write_secret_file.sh /tmp/pgbouncer_password

The skill reads these via --*-file flags and never embeds the value in rendered output.

cloud/automation-user.sh is idempotent when its destination already holds a private single-link mode-600 token: it exits without a REST mutation. A new mint or rotation requires the explicit non-secret gate SOAR_ACCEPT_TOKEN_MINT_OR_ROTATION=true; rotation also requires SOAR_ROTATE_AUTOMATION_TOKEN=true. The helper fsyncs a private pre-POST journal beside the token. An interrupted/uncertain POST, missing response key, or uncertain atomic write becomes ambiguous and blocks retries until an operator inspects the automation user and revokes any orphan token.

Quick Start

Render a single-instance unprivileged install:

SOAR_TGZ_SHA256="$(awk '{print $1}' /secure/vendor/splunk-soar.sha256)"
bash skills/splunk-soar-setup/scripts/setup.sh \
  --phase render \
  --soar-platform onprem-single \
  --soar-home /opt/soar \
  --soar-https-port 8443 \
  --soar-hostname soar01.example.com \
  --soar-tgz /tmp/splunk_soar-unpriv-8.5.0.tgz \
  --soar-tgz-sha256 "${SOAR_TGZ_SHA256}"

Render a 3-node cluster with external PostgreSQL on AWS RDS:

SOAR_TGZ_SHA256="$(awk '{print $1}' /secure/vendor/splunk-soar.sha256)"
bash skills/splunk-soar-setup/scripts/setup.sh \
  --phase render \
  --soar-platform onprem-cluster \
  --soar-home /opt/soar \
  --soar-https-port 8443 \
  --soar-hosts soar01,soar02,soar03 \
  --soar-ssh-known-hosts-file /secure/ssh/known_hosts \
  --soar-tgz /tmp/splunk_soar-unpriv-8.5.0.tgz \
  --soar-tgz-sha256 "${SOAR_TGZ_SHA256}" \
  --external-pg "mode=rds,host=soar-db.cluster-xyz.us-east-1.rds.amazonaws.com,port=5432" \
  --external-gluster gluster01,gluster02 \
  --external-es es01,es02,es03 \
  --load-balancer haproxy01

Render a SOAR Cloud onboarding bundle and Automation Broker install:

bash skills/splunk-soar-setup/scripts/setup.sh \
  --phase render \
  --soar-platform cloud \
  --soar-tenant-url https://example.splunkcloudgc.com/soar \
  --soar-automation-token-file /tmp/soar_automation_token \
  --automation-broker "runtime=docker,fips=auto"

Apply Splunk-side apps:

bash skills/splunk-soar-setup/scripts/setup.sh \
  --phase splunk-side-apps \
  --apply \
  --splunk-side-apps "app_for_soar=true,app_for_soar_export=true"

All mutating phases (apply, onprem-single, onprem-cluster, automation-broker, splunk-side-apps, and all) require the explicit --apply gate. Render, preflight, status, and validation remain non-mutating. Mutating on-prem package phases additionally require --soar-tgz-sha256 sourced from a trusted vendor or controlled artifact handoff. The wrapper validates that digest and the complete archive member graph before rendering or contacting a cluster host. Each installer extracts into a private validation stage before privileged host changes, requires the single top-level splunk-soar root and non-link installer entrypoints, then moves the validated root through a private same-filesystem promotion stage without replacing an existing installation. --phase es-integration exits nonzero before any mutation because the supported ES engine models conf-essoar as inventory/preflight-only. Complete tenant pairing, token entry, notable forwarding, and Adaptive Response verification in the documented Mission Control UI. Cluster provisioning, backup, and restore require a pre-enrolled SSH known-hosts file; generated scripts use strict host-key checking and never silently accept a new host key.

To mint the rendered Cloud automation token after review:

SOAR_ACCEPT_TOKEN_MINT_OR_ROTATION=true \
  bash splunk-soar-rendered/cloud/automation-user.sh

Do not delete an *.mint-state.json file in in_progress or ambiguous state to force a retry. Reconcile the live user's tokens and revoke any unknown credential first.

Validate Splunk-side SOAR apps (reads credentials from the project-root credentials file, checks that splunk_app_soar is installed, and prints a handoff hint for the SOAR UI):

bash skills/splunk-soar-setup/scripts/validate.sh \
  --soar-url https://soar01.example.com:8443

Add --export to also require the Splunk App for SOAR Export (Splunkbase 3411). For SOAR server-side health checks, run the rendered splunk-soar-rendered/validate.sh after the server-side install phases complete.

What It Renders

Under splunk-soar-rendered/:

• onprem-single/{prepare-system.sh, install-soar.sh, post-install-checklist.md}
• onprem-cluster/{make-cluster-node.sh, backup.sh, restore.sh}
• onprem-cluster/external-services/{postgres-rds.tf, postgres-local.sh, gluster-volume.sh, elasticsearch.yml, haproxy.cfg}
• shared/safe_extract_tar.py
• cloud/{onboarding-checklist.md, jwt-token-helper.sh, ip-allowlist.json, apply-allowlist.sh, automation-user.sh}
• automation-broker/{docker-compose.yml, podman-compose.yml, install.sh, add-ca-certificate.sh, preflight.sh}
• splunk-side/{install-app-for-soar.sh, install-app-for-soar-export.sh, configure-phantom-endpoint.sh}
• validate.sh

Out of Scope

• SOAR Cloud tenant provisioning (Splunk-managed; users get an invite from Splunk).
• Custom playbook authoring (use SCM/Git outside this skill).
• Vendor-specific SOAR connector apps (catalog-aware, not installed).
• SOAR licensing (commercial, separate from Splunk Enterprise licensing).
• Multi-tenant SOAR On-prem (Splunk Professional Services).
• SOAR Cloud → On-prem repatriation.

References

• reference.md for cluster topology, external-services setup, FIPS handling, REST automation user model, ES integration map, and backup/restore flow.
• template.example for the non-secret intake worksheet.

MCP Tools

This skill includes checked-in, read-only Splunk MCP custom tools generated from mcp_tools.source.yaml.

Validate or regenerate the tool artifact:

python3 skills/shared/scripts/mcp_tools.py validate skills/splunk-soar-setup
python3 skills/shared/scripts/mcp_tools.py generate skills/splunk-soar-setup

Load the tools into Splunk MCP Server:

bash skills/splunk-soar-setup/scripts/load_mcp_tools.sh

The loader uses the supported /mcp_tools REST batch endpoint by default. Use --allow-legacy-kv only for older MCP Server app versions that lack that endpoint.