Splunk Stream Setup

Splunk Stream Setup Automation

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Splunk Stream, stream forwarder, streamfwd, wire data, network capture, NetFlow, or packet capture setup.
• Preview and review the splunk stream setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

For a Windows capture host, delegate host investigation, transport, Npcap, Splunk_TA_stream, and transactional host validation to the child splunk-stream-windows-setup. The child supports local elevated PowerShell, Windows OpenSSH, WinRM, and AWS Systems Manager. Its read-only investigation and drift-bound plan are mandatory before Windows mutation. Return here for search/index-tier setup, protocol streams, indexed-data checks, _internal evidence, and dashboard completion.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-stream-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-stream-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

TA Completion Gate

For every TA/add-on or dashboard companion run, satisfy the shared TA completion gate: configure and enable the data ingest path owned by this skill or its required companion, validate events or metrics in the target indexes/source types, and verify any pre-built/package-shipped dashboards are visible, macro-aligned, and returning data. If the package ships no dashboards, record that evidence explicitly and hand off dashboard use to the consuming app, ES/ITSI/ARI content, or readiness doctor.

Automates installation and configuration of the Splunk Stream stack:

Package Model

On Splunk Enterprise, the install flow is:

1. try Splunkbase first for the current app ID
2. fall back to the original vendor archive in splunk-ta/ if Splunkbase is unavailable or the download fails

Current Splunkbase app IDs:

• splunk_app_stream: 1809
• Splunk_TA_stream: 5238
• Splunk_TA_stream_wire_data: 5234

Use the original vendor archives from splunk-ta/ as the local fallback packages:

• splunk-app-for-stream_816.tgz
• splunk-add-on-for-stream-forwarders_816.tgz
• splunk-add-on-for-stream-wire-data_816.tgz

For Splunk Cloud, the normal workflow is hybrid:

• install splunk_app_stream and Splunk_TA_stream_wire_data on the Cloud side using the original archives through ACS or the approved support workflow
• install Splunk_TA_stream unchanged on a customer-controlled HF/UF
• apply local HF overlay files such as templates/splunk-cloud-hf-netflow-any/

Any splunk-ta/_unpacked/ copy is review-only and not part of the normal deployment workflow.

Agent Behavior

The agent must NEVER ask for passwords or secrets in chat.

Splunk credentials are read automatically from the project-root credentials file (falls back to ~/.splunk/credentials). If neither exists, guide the user to create it:

bash skills/shared/scripts/setup_credentials.sh

The agent should still ask the user for non-secret configuration values:

• Stream forwarder IP address — the IP streamfwd listens on
• Stream forwarder port — default 8889
• Splunk Web URL — the full URL to Splunk Web
• SSL verification — whether streamfwd should verify SSL certs
• NetFlow configuration (optional) — receiver IP, port, decoder type
• Which protocol streams to enable
• Target index for stream data

Environment

This skill supports two different deployment patterns:

• Splunk Enterprise: either a single-instance workflow or a split-role workflow across search-tier, indexer, and forwarder targets. When SPLUNK_TARGET_ROLE is declared, the install path now scopes Stream package installation to the components modeled for that role. In that role-scoped mode, use explicit phase flags instead of the default full setup.
• Splunk Cloud: a hybrid workflow where the cloud search tier hosts splunk_app_stream, while Splunk_TA_stream runs on forwarders or hosts under your control. In Cloud mode, index creation uses ACS and the combined --install / --configure-streamfwd path is intentionally blocked.
• Windows capture tier: use splunk-stream-windows-setup. Prefer an x64 Universal Forwarder running as LocalSystem. The child blocks Stream capture in Splunk Enterprise 10.4+ on Windows because the platform's required least-privilege service identity conflicts with Stream's documented Windows service-account requirement.

Remote Splunk Connection

To run against a remote Splunk instance:

export SPLUNK_SEARCH_API_URI="https://splunk-host:8089"

Available Packages

Setup Workflow

Step 1: Install Apps

bash skills/splunk-stream-setup/scripts/setup.sh --install

On Splunk Enterprise, installs any of the three apps that are not already present. It uses the splunk-app-install skill's install_app.sh under the hood and follows the same package policy as the rest of the repo: Splunkbase first, then local fallback from splunk-ta/. For local fallback installs on a remote host, the installer uses the same remote-host behavior as the generic app installer: stage the package over SSH, then install the staged server-local path through the management API with filename=true.

When SPLUNK_TARGET_ROLE is set, the install step uses the deployment-role matrix to scope the package set:

• search-tier: splunk_app_stream plus search-tier-compatible Stream support
• indexer: Splunk_TA_stream_wire_data
• heavy-forwarder: Splunk_TA_stream plus forwarder-compatible Stream support
• universal-forwarder: Splunk_TA_stream

When no role is declared, the normal setup path now fails fast instead of guessing an all-in-one placement. Use --legacy-all-in-one only when you explicitly want the older single-target behavior.

On Splunk Cloud, do not run the combined install path against the cloud search tier. Splunk documents Stream on Cloud as a hybrid deployment:

• splunk_app_stream is provisioned on the Splunk Cloud search tier with help from your account team / support workflow
• Splunk_TA_stream runs on forwarders or hosts under your control
• Splunk_TA_stream_wire_data should be installed according to the Stream cloud deployment architecture approved for your environment

Step 2: Create Indexes

bash skills/splunk-stream-setup/scripts/setup.sh --indexes-only

In Splunk Cloud, --indexes-only creates these indexes through ACS.

Step 3: Configure Stream Forwarder

bash skills/splunk-stream-setup/scripts/setup.sh \
  --configure-streamfwd \
  --ip-addr "<capture-host-ip>" \
  --port 8889 \
  --splunk-web-url "https://<stream-search-tier-host>:8000" \
  --ssl-verify false

Use the capture host's address for --ip-addr and the reachable Stream search-tier/app endpoint for --splunk-web-url; these values are deployment specific and must not be copied from an example.

Writes local/streamfwd.conf and local/inputs.conf in the Stream TA. For Splunk Cloud, run this step against the forwarder-side Splunk instance you control, not the cloud search tier. Use the cloud-hosted Stream app URL on 443 or 8443 as provided by your Splunk Cloud deployment.

Optional NetFlow receiver:

bash skills/splunk-stream-setup/scripts/setup.sh \
  --configure-streamfwd \
  --ip-addr "<capture-host-ip>" \
  --port 8889 \
  --splunk-web-url "https://<stream-search-tier-host>:8000" \
  --ssl-verify false \
  --netflow-ip "0.0.0.0" \
  --netflow-port 9995 \
  --netflow-decoder netflow

Step 4: Enable Protocol Streams

bash skills/splunk-stream-setup/scripts/configure_streams.sh \
  --enable dns,http,tcp,udp,dhcp,netflow \
  --index main

The script manages stream definitions through the Stream KV Store or Stream Web API when available. If neither endpoint is reachable, enable or disable streams from Splunk Web instead.

Available protocols: amqp, arp, dhcp, diameter, dns, ftp, http, icmp, igmp, imap, ip, irc, ldap, mapi, modbus, mysql, netflow, nfs, pop3, postgres, radius, rtcp, rtp, sflow, sip, smb, smpp, smtp, snmp, tcp, tds, tns, udp, xmpp.

Aggregated (Splunk_*) streams: Splunk_DNSClientErrors, Splunk_DNSClientQueryTypes, Splunk_DNSIntegrity, Splunk_DNSRequestResponse, Splunk_DNSServerErrors, Splunk_DNSServerQuery, Splunk_DNSServerResponse, Splunk_HTTPClient, Splunk_HTTPResponseTime, Splunk_HTTPStatus, Splunk_HTTPURI, Splunk_IP, Splunk_MySql, Splunk_Postgres, Splunk_SSLActivity, Splunk_Tcp, Splunk_Tds, Splunk_Tns, Splunk_Udp.

Step 5: Restart If Required

On Splunk Enterprise, restart Splunk after app or index changes. On Splunk Cloud, check acs status current-stack and only run acs restart current-stack when ACS reports restartRequired=true.

Step 6: Validate

bash skills/splunk-stream-setup/scripts/validate.sh

Checks the cloud search tier or Enterprise target that SPLUNK_SEARCH_API_URI (or legacy SPLUNK_URI) points to. In Splunk Cloud, forwarder-side Splunk_TA_stream checks are reported as non-fatal warnings because that component usually lives outside the cloud search tier.

Key Configuration Files

Deployment Templates

For a customer-controlled heavy forwarder sending NetFlow/IPFIX to Splunk Cloud, use:

skills/splunk-stream-setup/templates/splunk-cloud-hf-netflow-any/

That template set includes:

• Splunk_TA_stream/local/inputs.conf
• Splunk_TA_stream/local/streamfwd.conf
• system/local/outputs.conf

The template is intentionally open to NetFlow/IPFIX from any device and is meant to be copied onto the heavy forwarder after you replace the placeholder values.

Sourcetypes

All stream sourcetypes follow the pattern stream:<protocol>:

Known Issues

1. Cloud deployment is hybrid: Splunk Stream on Splunk Cloud uses a cloud-hosted splunk_app_stream plus forwarders you control. Do not treat it as a single-target install.
2. Install order matters on Enterprise: Install splunk_app_stream first, then Splunk_TA_stream, then Splunk_TA_stream_wire_data.
3. Restart behavior differs by platform: Enterprise restarts Splunk directly. Splunk Cloud uses ACS restart checks.
4. No sudo needed: Scripts run as the splunk OS user.
5. SSL verification: Set sslVerifyServerCert = false in inputs.conf for self-signed certs.
6. Stream forwarder permissions: streamfwd may need packet-capture capabilities such as cap_net_raw on the Splunk host. Apply those host-side permissions using your standard deployment process if raw capture is required.
7. KV Store: Stream app uses KV Store for stream definitions. Ensure KV Store is healthy before configuration.

Additional Resources

• reference.md — Package IDs, deployment roles, indexes, ports, validation