Splunk Indexer Cluster Setup

Splunk Indexer Cluster Setup

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Bootstrap an indexer cluster, configure site_replication_factor or site_search_factor, apply or roll back a cluster bundle, perform searchable rolling restarts, take a peer offline, migrate single-site to multisite, decommission a site, or.
• Preview and review the splunk indexer cluster setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-indexer-cluster-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

This skill sits above skills/splunk-enterprise-host-setup, which still owns per-host install/upgrade. It owns multi-host orchestration of the cluster control plane and every documented cluster operation.

Architecture First

• The host-setup skill installs Splunk Enterprise per host. This skill configures those installed hosts as a coordinated cluster (single-site or multisite).
• Cluster manager redundancy uses two or more managers in active/standby; the skill renders the LB + DNS recipes per Splunk's documented patterns.
• Multisite migration keeps both legacy replication_factor/search_factor AND the new site_* factors so existing buckets remain valid.

Agent Behavior — Credentials

Never paste secrets into chat.

• Use template.example for the non-secret intake worksheet (manager URI, peer/SH lists, factors, sites).
• Keep secrets in temporary files only:

bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_admin_password
bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_idxc_secret

The cluster pass4SymmKey is distinct from the license manager pass4SymmKey and any SHC pass4SymmKey.

Quick Start

Single-site bootstrap (3 peers, 1 SH, 1 manager):

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase render \
  --cluster-mode single-site \
  --cluster-label prod \
  --cluster-manager-uri https://cm01.example.com:8089 \
  --manager-hosts cm01.example.com \
  --replication-factor 3 \
  --search-factor 2 \
  --peer-hosts idx01.example.com,idx02.example.com,idx03.example.com \
  --sh-hosts sh01.example.com

Multisite bootstrap with explicit per-site factors:

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase render \
  --cluster-mode multisite \
  --cluster-label prod \
  --cluster-manager-uri https://cm01.example.com:8089 \
  --manager-hosts cm01.example.com \
  --available-sites site1,site2 \
  --site-replication-factor "origin:2,total:3" \
  --site-search-factor "origin:1,total:2" \
  --peer-hosts "idx01.example.com=site1,idx02.example.com=site2" \
  --sh-hosts "sh01.example.com=site1"

Apply a cluster bundle (validates, then applies):

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase bundle-apply \
  --cluster-manager-uri https://cm01.example.com:8089 \
  --admin-password-file /tmp/splunk_admin_password

Searchable rolling restart with health check:

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase rolling-restart \
  --rolling-restart-mode searchable \
  --cluster-manager-uri https://cm01.example.com:8089 \
  --admin-password-file /tmp/splunk_admin_password

Take a peer offline (fast):

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase peer-offline \
  --peer-offline-mode fast \
  --peer-host idx02.example.com \
  --admin-password-file /tmp/splunk_admin_password

Migrate a single-site cluster to multisite:

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase migrate-to-multisite \
  --cluster-manager-uri https://cm01.example.com:8089 \
  --available-sites site1,site2 \
  --site-replication-factor "origin:2,total:3" \
  --site-search-factor "origin:1,total:2" \
  --site-mappings "default_mapping:site1"

Run targeted migration/recovery operations through the wrapper:

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase replace-manager \
  --cluster-manager-uri https://cm01.example.com:8089 \
  --new-manager-uri https://cm02.example.com:8089 \
  --idxc-secret-file /tmp/splunk_idxc_secret \
  --admin-password-file /tmp/splunk_admin_password

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase decommission-site \
  --cluster-manager-uri https://cm01.example.com:8089 \
  --site site2 \
  --accept-site-decommission \
  --admin-password-file /tmp/splunk_admin_password

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase move-peer-to-site \
  --cluster-manager-uri https://cm01.example.com:8089 \
  --peer-host idx02.example.com \
  --new-site site2 \
  --admin-password-file /tmp/splunk_admin_password

bash skills/splunk-indexer-cluster-setup/scripts/setup.sh \
  --phase migrate-non-clustered \
  --cluster-manager-uri https://cm01.example.com:8089 \
  --indexer-host idx03.example.com \
  --idxc-secret-file /tmp/splunk_idxc_secret \
  --admin-password-file /tmp/splunk_admin_password

What It Renders

Under splunk-indexer-cluster-rendered/cluster/:

• manager/<host>/server.conf — primary + (optional) standby manager configs.
• peer-<host>/server.conf — per-peer config with site assignment.
• sh-<host>/server.conf — per-SH config with multisite/affinity.
• bootstrap/sequenced-bootstrap.sh — manager → peers (RF gate) → SHs.
• bundle/{validate.sh, status.sh, apply.sh, apply-skip-validation.sh, rollback.sh}.
• restart/{rolling-restart.sh, searchable-rolling-restart.sh, force-searchable.sh}.
• maintenance/{enable.sh, disable.sh}.
• peer-ops/{offline-fast.sh, offline-enforce-counts.sh, remove-peer.sh, extend-restart-timeout.sh}.
• redundancy/{lb-haproxy.cfg, dns-record-template.txt, ha-health-check.sh} (when redundancy enabled).
• migration/{single-to-multisite.sh, replace-manager.sh, decommission-site.sh, move-peer-to-site.sh, migrate-non-clustered.sh}.
• forwarder-outputs/<host>/outputs.conf — indexer-discovery snippets for HF/UF.
• validate.sh — REST cluster-manager audit snapshot (info/health/peers/sites/buckets/generation/status + bundle status) saved under audit/<timestamp>/, then gates on the /services/cluster/manager/info preflight_check_passed field (PASS/FAIL).

Hand-off Contracts

• Assumes hosts are installed by splunk-enterprise-host-setup --phase install.
• Emits a LICENSE_PEERS[] stub at splunk-indexer-cluster-rendered/cluster/handoffs/license-peers.txt so splunk-license-manager-setup can wire up the license peer config.
• Warns when bundle apps include SmartStore-aware files and points to skills/splunk-index-lifecycle-smartstore-setup for indexes.conf rendering.
• For Search Head Cluster setup (deployer, SHC members, rolling restarts, captain transfer, KV Store), see splunk-search-head-cluster-setup.

Out of Scope

• Cluster pass4SymmKey rotation: rendered server.conf files contain pass4SymmKey = $IDXC_SECRET so the operator can manage the secret out of band (env var, secrets manager, splunk hash-passwd). Rotating the secret cluster-wide remains a manual rolling restart with the new value; this skill does not orchestrate that rolling rotation.
• Manager DR backup / restore: backup/restore of master-apps/ and manager state is operator-owned. The redundancy templates render active/standby manager pairs but do not snapshot or replay manager state.
• Splunk Cloud indexer clusters: Splunk-managed; this skill targets self-managed Splunk Enterprise only.

References

• reference.md for full multisite semantics, redundancy topologies, bundle reload-vs-restart classification, and rolling-restart health-check details.
• template.example for the non-secret intake worksheet.