Splunk MCP Server Setup

Splunk MCP Server Setup

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Splunk MCP server setup, Splunk MCP TA, Splunk_MCP_Server, /services/mcp, the hosted SCS MCP Gateway for Splunk Observability Cloud, Cursor MCP, Codex MCP, Claude Code MCP connectivity to Splunk, or Cisco Data Fabric agentic/tool access.
• Preview and review the splunk mcp server setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-mcp-server-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-mcp-server-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Production release status

The current official package, 1.3.1, is not production-approved by this repository's security and protocol review. Installation fails closed by default while vendor fixes are pending. Do not deploy it to production or expose it to untrusted clients. --accept-nonproduction-package exists only for isolated evaluation and does not make the package production-safe.

Version 1.3.1 is the supported evaluation baseline. It includes the 1.3 SPL2 @spl2 query path, custom-tool UI, find_data_source preview, rest command support, role-to-tool mappings, allowed-command administration, and workload-pool selection. Treat those as vendor features: this skill validates their shipped administration surfaces but does not silently enable tools, expand allowed SPL commands, or assign roles.

Shared add-on completion gate

Whenever this workflow installs, configures, or hands off the packaged MCP app, follow the shared completion gate. Package delivery alone is not success; validate the configured endpoint and its data access, plus any shipped views, or record explicit package evidence that no dashboards ship.

Automates setup of the Splunk MCP Server app (Splunk_MCP_Server).

For newer Cisco Data Fabric wording, this is the MCP/tool-access route for agentic workflows. Federated search, data pipelines, and AI Toolkit workflows remain in their dedicated skills.

What This Skill Covers

This skill handles five operator tasks:

1. Install or update the packaged app from the repo-local splunk-ta/ cache
2. Configure supported runtime settings in mcp.conf
3. Mint encrypted bearer tokens into local-only files
4. Render a reusable local bridge bundle for Cursor, Codex, and Claude Code, targeting either local Splunk Platform /services/mcp or the hosted SCS MCP Gateway
5. Uninstall the app cleanly when lab teardown is needed

The bridge bundle uses the same mcp-remote wrapper pattern for all three tools, so one rendered directory can be opened in Cursor, registered with Codex, and auto-wired into Claude Code's .mcp.json. The wrapper passes header placeholders such as ${SPLUNK_MCP_HEADER_X_SF_TOKEN} to mcp-remote, keeping token values in the local env file instead of command argv.

mcp-remote is an experimental compatibility proxy. Prefer a client's native Streamable HTTP transport when it can supply the required headers; the pinned bridge is not itself a production trust boundary.

Package Model

Use the repo-local package in splunk-ta/ as the default install source.

The packaged app currently lives in:

splunk-ta/splunk-mcp-server_131.tgz

The setup workflow is the required install path because it enforces package provenance and the production-review gate. For isolated evaluation only:

bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --install \
  --accept-nonproduction-package

To remove the app again:

bash skills/splunk-mcp-server-setup/scripts/setup.sh --uninstall

Agent Behavior — Credentials And Tokens

The agent must NEVER ask for passwords, bearer tokens, or other secrets in chat.

Splunk credentials come from the project-root credentials file (falls back to ~/.splunk/credentials):

bash skills/shared/scripts/setup_credentials.sh

MCP bearer tokens are secrets. Always write them to a local-only file:

bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --token-user "${SPLUNK_USER}" \
  --write-token-file /tmp/splunk_mcp.token

The agent may freely ask for non-secret values such as:

• MCP token username
• desired token lifetime
• row limits
• rate-limit thresholds
• whether the rendered client bridge should assume insecure TLS for lab use
• hosted SCS region, Observability realm, and Splunk tenant name

Use an existing Splunk user that has the mcp_tool_admin capability. In most lab setups that should be the same account already configured in SPLUNK_USER.

For prerequisite collection, use skills/splunk-mcp-server-setup/template.example as the intake worksheet and keep any filled copy local as template.local.

Environment

Setup Workflow

Step 1: Install Or Update The App

bash skills/splunk-mcp-server-setup/scripts/setup.sh --install

For an isolated Enterprise lab where service interruption should be deferred:

bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --install \
  --no-restart \
  --accept-nonproduction-package

--no-restart only defers the shared installer restart; it does not guarantee that updated Python handlers are active. Run validation afterward. If the live endpoint still exposes the prior app behavior, restart Splunk before relying on the upgrade.

This command intentionally refuses the current review-blocked release. After a fixed vendor release is reviewed and marked production-approved in package-manifest.json, the setup script will detect install versus update mode automatically. Use --accept-nonproduction-package only in an isolated lab.

Alternative: Uninstall The App

bash skills/splunk-mcp-server-setup/scripts/setup.sh --uninstall

This delegates to the shared app uninstaller for Splunk_MCP_Server and restarts Splunk automatically on Enterprise targets unless the shared workflow is changed to skip restart. Run --uninstall by itself; do not combine it with render, token, or configuration flags.

Step 2: Configure Supported MCP Server Settings

bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --timeout 90 \
  --max-row-limit 2000 \
  --default-row-limit 250 \
  --ssl-verify true \
  --require-encrypted-token true \
  --legacy-token-grace-days 0 \
  --token-default-lifetime-seconds 43200 \
  --token-max-lifetime-seconds 86400 \
  --token-key-reload-interval-seconds 300 \
  --global-rate-limit 600 \
  --admission-global 60 \
  --tenant-authenticated 240 \
  --tenant-unauthenticated 10 \
  --circuit-breaker-failure-threshold 5 \
  --circuit-breaker-cooldown-seconds 60

This updates supported fields in mcp.conf:

• [server] timeout
• [server] max_row_limit
• [server] default_row_limit
• [server] ssl_verify (configuration intent only; vendor 1.3.1 does not enforce it for internal HTTP calls)
• [server] require_encrypted_token
• [server] legacy_token_grace_days
• [server] mcp_token_default_lifetime_seconds
• [server] mcp_token_max_lifetime_seconds
• [server] token_key_reload_interval_seconds
• [rate_limits] admission and circuit-breaker values

The script also fixes visible=true on the app if ACS or local installs left it hidden in Splunk Web.

Step 3: Optionally Rotate The MCP RSA Keys

bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --rotate-keys \
  --rotate-key-size 4096

Step 4: Mint An Encrypted Bearer Token

bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --token-user "${SPLUNK_USER}" \
  --token-expires-on +12h \
  --write-token-file /tmp/splunk_mcp.token

The script writes the encrypted token to the target file with 0600 permissions. It does not print the token to stdout.

Release 1.3.1 mints tokens with an authenticated GET /mcp_token request; POST /mcp_token is reserved for action=rotate. The helper URL-encodes the username and lifetime fields required by that vendor contract. Those non-secret fields can appear in Splunk or intermediary access logs, but the returned token is kept out of URLs and written only to the requested mode-0600 file.

If you disable require_encrypted_token, the app intentionally fails closed on /mcp_token minting and key rotation. Do not combine --require-encrypted-token false with --write-token-file or --rotate-keys in the same run.

Step 5: Render And Apply The Shared Cursor/Codex Bridge Bundle

Choose one gateway mode:

Default platform mode preserves the existing local app workflow:

bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --render-clients \
  --bearer-token-file /tmp/splunk_mcp.token \
  --cursor-workspace ~/Projects/my-cursor-workspace

O11y-only hosted gateway:

bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --render-clients \
  --gateway-mode o11y \
  --scs-region pdx10 \
  --o11y-realm us1 \
  --o11y-token-file /tmp/splunk_o11y_api_token

Combined Splunk Platform + Observability gateway:

bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --render-clients \
  --gateway-mode combined \
  --scs-region pdx10 \
  --o11y-realm us1 \
  --o11y-token-file /tmp/splunk_o11y_api_token \
  --splunk-tenant mytenant \
  --splunk-jwt-file /tmp/splunk_mcp_jwt

The SCS gateway URL is derived as:

https://region-<REGION>.api.scs.splunk.com/system/mcp-gateway/v1/

Current documented realm-to-SCS-region mappings:

Google Cloud Platform realms and GovCloud realms are not supported by the hosted MCP Gateway; the renderer rejects known unsupported values such as us2 and gov*. Use --gateway-url only when Splunk provides an explicit gateway endpoint.

Default render target:

./splunk-mcp-rendered

The rendered bundle contains:

• .cursor/mcp.json for Cursor
• run-splunk-mcp.sh as a shell stdio-to-HTTP bridge
• run-splunk-mcp.js as the Node bridge used by Cursor, Codex, and Claude Code registrations
• .env.splunk-mcp.example
• .env.splunk-mcp when a token file is supplied
• register-codex-mcp.sh to sync a portable Codex launcher bundle under ~/.codex/mcp-bridges/

When --render-clients runs, the skill also applies client setup by default:

• registers CLIENT_NAME with Codex using a stable home-local launcher copy so repo moves do not break startup
• merges the Splunk MCP entry into <cursor-workspace>/.cursor/mcp.json
• writes the Splunk MCP entry into <workspace>/.mcp.json for Claude Code
• defaults the workspace target to the current working directory when --cursor-workspace is omitted

Use --no-register-codex, --no-configure-cursor, or --no-configure-claude to opt out of any client update while still rendering the bundle.

Both wrappers require an operator-installed mcp-remote@0.1.38 on PATH, verify that exact package version, and fail closed rather than downloading code at startup. For o11y and combined gateway modes, the wrapper also passes --transport http-only --allow-http to match Splunk's hosted gateway examples.

Do not add hosted Observability AI Assistant MCP tools to local Splunk_MCP_Server custom tool manifests. Gateway mode only configures client headers and endpoint selection.

Step 6: Validate

bash skills/splunk-mcp-server-setup/scripts/validate.sh \
  --completion \
  --mcp-bearer-token-file /tmp/splunk_mcp.token

Checks:

• app installed and visible
• installed app version exactly matches the reviewed manifest (and is at least 1.3.1)
• /services/mcp completes authenticated initialize, notifications/initialized, tools/list, and a safe splunk_get_info tool call
• /services/mcp rejects an untrusted Origin
• enabled tools exactly match the reviewed allowlist (minimal default: ["splunk_get_info"]; use --allowed-tools-file for a reviewed expansion)
• key MCP REST endpoints respond, including the 1.3 tool-role, guardrail, and allowed-SPL-command administration surfaces
• protected-resource metadata endpoint is reachable when configured
• current server settings and rate-limit values are readable
• encrypted-token, zero-grace, short-lifetime, and nonzero admission settings meet the production policy
• ssl_verify is reported as configuration-only and release 1.3.1 fails because the vendor does not enforce it
• derived /services/mcp URL is sane
• the shipped dashboard, monitoring, tools, and tool_settings views are visible

Policy Surfaces In 1.3.1

The package still ships these baseline policy files with local-over-default precedence:

• local/safe_spl.json
• local/generating_commands.json

Version 1.3.1 also ships authenticated administration endpoints for allowed SPL commands, tool-role mappings, and guardrails. This skill validates that those surfaces exist but does not mutate them. Review proposed commands and role assignments separately, use the Splunk UI or documented app endpoint, and then pass the exact enabled-tool policy to validation with --allowed-tools-file. On self-managed targets, app-local overlays remain available when filesystem ownership is explicit. On Splunk Cloud, treat direct file changes as package content and use supported administration surfaces.

See reference.md for the exact implications.

Key Learnings / Known Issues

1. safe_spl.json remains the local baseline: the app loads it from the app directory; the 1.3 allowed-command endpoint manages a separate REST-backed command layer rather than rewriting that JSON file.
2. Token output is secret material: write encrypted bearer tokens to local files, never to chat or tracked repo files.
3. The shared wrapper is the most portable client path: Cursor, Codex, and Claude Code can all use the rendered run-splunk-mcp.js bridge via mcp-remote.
4. mcp.conf is the supported remote configuration surface: use it for row limits and token policy. Release 1.3.1 does not enforce its documented ssl_verify value, so that field cannot satisfy a TLS control.
5. The app needs search-tier placement: it exposes /services/mcp and depends on custom REST handlers plus KV Store-backed tool metadata.
6. Hosted SCS MCP Gateway is client-side configuration: it uses --gateway-mode o11y or combined and does not install hosted Observability tools into the local Splunk Platform app.
7. Safe-SPL exclusion is defective in 1.3.1: never use exclude_tools to disable a tool; use mcp_tools_enabled, and keep query tools away from untrusted callers pending a vendor fix.
8. Evaluation data must be synthetic: 1.3.1 logs tool arguments and SPL to _internal. Never embed literal credentials in custom tool headers or bodies.

Client Activation

For Cursor and Claude Code activation details, including the local-only token file and reload requirements, read the client activation notes.

Additional Resources

• reference.md — endpoint map, config surface, and client notes
• template.example — non-secret intake worksheet