Splunk Observability Claude Code Instrumentation Setup

Splunk Observability Claude Code Instrumentation Setup

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Instrumenting Claude Code to emit metrics, log events, and distributed traces (beta) to Splunk Observability Cloud via a local OTel Collector fan-out, with optional Galileo OTLP trace ingestion for AI observability; covers all three.
• Preview and review the splunk observability claude code instrumentation setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Overview

Claude Code has native OpenTelemetry support. Metrics, log events, and traces (beta) are configured entirely through environment variables and the .claude/settings.json env block. This skill renders those configuration assets, an optional local OTel Collector overlay, and a otelHeadersHelper shim for secret-safe direct-mode authentication.

Required Intake

Before enabling Galileo, ask the user for the exact Galileo instance console URL and use the value they provide, for example https://console.demo-v2.galileocloud.io/. Do not assume the public Galileo Cloud tenant. Pass the answer as --galileo-console-url; an explicitly supplied --galileo-otel-endpoint is also accepted when the deployment does not follow Galileo's documented console-to-API hostname convention. The renderer fails closed when Galileo is enabled without either URL.

Claude Code exposes exactly one global OTEL_EXPORTER_OTLP_HEADERS value. That means the CLI itself cannot fan out to two destinations that require different auth headers. To send telemetry to both Splunk Observability Cloud and Galileo Observe at the same time, Claude Code must ship to a local OTel Collector, and the collector must fan out. The skill defaults to that mode.

The skill renders three destination modes:

• local-collector (default): Claude Code emits OTLP to http://127.0.0.1:14318. A rendered collector overlay exports metrics via the SignalFx exporter (send_otlp_histograms: true), traces via OTLP APM ingest, and fans traces out to Galileo Observe when Galileo is enabled.
• splunk-direct: Claude Code emits OTLP/HTTP directly to https://ingest.<realm>.observability.splunkcloud.com/v2/{trace,datapoint,log}/otlp, authenticated by a otelHeadersHelper script that reads the token from SPLUNK_O11Y_TOKEN_FILE. No collector is required. Galileo is not reachable in this mode.
• external-collector: Operator-specified OTLP endpoints; the collector overlay is not rendered. Header values must be safe literals or environment placeholders.

Galileo is optional for collector-capable destinations (local-collector, external-collector, all). Passing --galileo-project enables it automatically; pass --galileo-enabled when using a spec-driven render that already contains the project/log-stream values. Use --disable-galileo for a Splunk-only render. --galileo-project is required whenever Galileo is enabled, and the user-confirmed instance URL is required regardless of project or log-stream values.

Note on logs: the Splunk Observability logs OTLP path is included in the overlay, but Splunk Observability Cloud ingests logs through Log Observer / HEC rather than the O11y OTLP logs endpoint. Treat the O11y logs pipeline as best-effort; route Claude Code log events to Splunk Platform via HEC when you need them searchable.

Note on where data appears: Claude Code metrics land as the native claude_code.* namespace — find them under Metrics → Metric Finder (search claude_code). The prebuilt Splunk "AI overview" (AI Agent Monitoring) dashboard instead reads GenAI-convention APM spans, so the local-collector overlay includes a transform/claude_code_genai processor that maps Claude Code's llm_request spans to gen_ai.operation.name=chat + gen_ai.usage.* + Client span kind, stamps gen_ai.agent.name onto Claude spans, and marks root claude_code.interaction spans as gen_ai.operation.name=invoke_workflow. The overlay derives gen_ai.client.operation.duration in seconds with a spanmetrics connector. Claude Code's reliable native claude_code.token.usage sum is first normalized, converted from cumulative to delta when necessary, and observed by signal_to_metrics/claude_code_token_histogram as the required gen_ai.client.token.usage histogram. The old sum-connector path is invalid for the prebuilt Tokens/Cost tiles because it creates a counter with the right name but the wrong metric type. The rendered settings and collector also stamp sf_environment because the AI overview Environment picker filters on Splunk's sf_environment, not only OTel deployment.environment. The derived GenAI metrics are exported through Splunk OTLP metric ingest so those Splunk dimensions are preserved. These transforms run only in collector modes; splunk-direct cannot feed the AI overview.

When Galileo fan-out is enabled, a second, Galileo-only transform/claude_code_galileo maps Claude's detailed user_prompt, new_context, and response.model_output attributes to OpenInference input/output fields. It also promotes the parent claude_code.tool span to an execute_tool operation, copies tool arguments/results, advertises Claude's tools inventory on LLM spans, and filters duplicate permission/execution children. This transform must run after transform/claude_code_genai and before the Galileo filter. It is intentionally absent from the Splunk trace branch.

The token histogram requires a collector build containing the alpha signal_to_metrics connector. otel/opentelemetry-collector-contrib:0.158.0 is validated with this overlay. The stock Splunk Distribution 0.158.0 still does not include that connector, even though the rest of the overlay starts there (otelcol validate rejects the overlay on the Splunk build with unknown type: "signal_to_metrics"). Use a matching contrib build or a custom collector that includes the connector; do not silently fall back to a sum connector. Start the collector before the new Claude process so cumulative-to-delta can retain the first counter value using its normal gateway heuristic.

Diagnostic boundary: if Splunk contains traces.count for chat <model>, a HISTOGRAM gen_ai.client.operation.duration, and a HISTOGRAM gen_ai.client.token.usage under the intended sf_environment, but the AI overview's internal count(agents) stream remains zero, the collector path is healthy. Check Settings -> AI agent monitoring and confirm the organization stores AI conversation data in Splunk Observability Cloud. The AI overview is not supported when that data source is set to Splunk logs; this is an organization-level product setting, not a Claude Code exporter failure.

Galileo fan-out uses a separate filtered trace pipeline. Splunk receives full Claude traces; Galileo receives only spans with GenAI semantic-convention attributes so root workflow spans do not create partialSuccess warnings.

Shared-collector warning: when Claude Code shares one OTLP receiver with Codex or other agents, do not route Claude signals only by resource-level service.name or data.source. Claude Code can place those identifiers on spans and metric datapoints instead of the resource envelope. Use the rendered runtime/shared-collector-routing.md pattern: context: span for traces, context: metric / context: datapoint for metrics, and context: log for logs.

For a Docker collector, http://127.0.0.1:14318 is Claude's host-side client endpoint, not the receiver bind address inside the container. Publish 127.0.0.1:14318:4318, pass --collector-receiver-endpoint 0.0.0.0:4318 for a standalone rendered overlay, or merge the processors, connectors, exporters, and routes into an existing receiver bound to 0.0.0.0:4318.

Traces are a Claude Code beta. They require OTEL_TRACES_EXPORTER=otlp plus CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1. Base and detailed beta tracing are separate controls. Base traces are on by default; detailed tracing is off because it can emit experimental content-bearing attributes. Enable it only with --enable-detailed-traces --accept-content-capture. Current Claude Code emits claude_code.llm_request and claude_code.tool under base beta tracing; detailed tracing adds claude_code.hook and experimental content-bearing span attributes. Interactive detailed tracing can require Anthropic allowlisting.

Detailed Tracing and Galileo Ingest

Detailed tracing requires its separate endpoint variable and explicit content acceptance; base beta tracing still owns current LLM and tool spans. Galileo accepts only spans carrying gen_ai.* attributes. Use the exact variables, diagnostic order, and ingest contract in reference.md and reference.md.

Safety Rules

• Never pass a Splunk access token, Galileo API key, or any secret on argv. Reject direct secret flags including equals form: --token, --access-token, --sf-token, --o11y-token, --api-key, --galileo-api-key, and --password.
• Direct-mode auth is delivered through otelHeadersHelper, a top-level settings.json key pointing to a script that reads the token from SPLUNK_O11Y_TOKEN_FILE and prints the OTLP headers as JSON. The literal token value never lands in settings.json, in env blocks, or in argv.
• In external-collector OTLP/HTTP mode, placeholder-backed headers such as Authorization=${OTLP_AUTH} are also resolved by otelHeadersHelper from the Claude process environment. Dynamic headers are unsupported by Claude's gRPC exporter, so the renderer rejects unresolved gRPC placeholders.
• Galileo API keys live in GALILEO_API_KEY_FILE. The collector overlay reads the value at collector process start through ${env:GALILEO_API_KEY}, which is populated by an operator-owned wrapper that sources the file.
• Content capture is off by default. Enabling any of OTEL_LOG_USER_PROMPTS=1, OTEL_LOG_ASSISTANT_RESPONSES=1, OTEL_LOG_TOOL_DETAILS=1, OTEL_LOG_TOOL_CONTENT=1, or OTEL_LOG_RAW_API_BODIES requires --accept-content-capture. Raw API bodies contain the conversation history; use file:/absolute/directory only after reviewing local retention and permissions.
• Detailed beta tracing also requires --accept-content-capture because its experimental span attributes can include prompt, tool, or model content.
• The skill refuses to render Galileo assets for splunk-direct (Claude Code cannot send two independent auth headers).
• --apply consumes the reviewed apply-plan.json already present in --output-dir. If no apply plan exists, the skill renders from the current options first.

Destinations

Primary Workflow

After collecting the required instance URL, render local collector assets with Splunk + Galileo fan-out. Passing --galileo-project enables Galileo automatically. The skill derives the OTLP endpoint from documented app.galileo.ai, console.<tenant>, and console-<tenant> URL forms:

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/setup.sh \
  --render \
  --destination local-collector \
  --local-collector-endpoint http://127.0.0.1:14318 \
  --collector-receiver-endpoint 0.0.0.0:4318 \
  --realm us1 \
  --galileo-console-url https://console.demo-v2.galileocloud.io/ \
  --galileo-project coding-agents \
  --galileo-log-stream claude-code \
  --output-dir splunk-observability-claude-code-instrumentation-rendered

For public Galileo Cloud, pass the user-confirmed --galileo-console-url https://app.galileo.ai/; it derives https://api.galileo.ai/otel/traces. Base traces are on by default, so no --enable-traces-beta flag is required. Detailed tracing remains off unless explicitly enabled with content-capture acceptance.

Render direct Splunk Observability metrics, logs, and traces:

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/setup.sh \
  --render \
  --destination splunk-direct \
  --realm us1 \
  --enable-traces-beta

Render both destinations side by side:

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/setup.sh \
  --render \
  --destination all \
  --realm us1

Render an external OTLP collector profile. Provide either a single --external-collector-endpoint (used as the base for all signals) or explicit per-signal endpoints (--external-trace-endpoint, --external-metric-endpoint, --external-log-endpoint). When traces beta is on, a trace endpoint is required (either the shared base or the explicit trace endpoint):

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/setup.sh \
  --render \
  --destination external-collector \
  --external-collector-endpoint https://otel-gateway.example.com:4318 \
  --external-collector-protocol http/protobuf

No collector overlay is rendered in this mode — the operator owns the collector.

Validate rendered output:

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/validate.sh \
  --output-dir splunk-observability-claude-code-instrumentation-rendered

For a shared deployment, validate the actual merged collector configuration as well as the rendered assets:

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/validate.sh \
  --output-dir splunk-observability-claude-code-instrumentation-rendered \
  --collector-config ~/.config/otelcol/config.yaml

Apply only after review:

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/setup.sh \
  --apply settings \
  --settings-scope user

Preview apply operations without writing:

bash skills/splunk-observability-claude-code-instrumentation-setup/scripts/setup.sh \
  --apply all \
  --dry-run \
  --json

Rendered Artifacts

• settings/claude-settings.<scope>.<destination>.json: the rendered settings.json fragment containing an env block with CLAUDE_CODE_ENABLE_TELEMETRY, OTEL_* exporter selections, per-signal endpoints, cardinality flags, and (for direct or dynamic external HTTP auth) the top-level otelHeadersHelper key.
• env/claude-code-o11y.<destination>.env: a shell-source-friendly copy of the same env block for operators who prefer to export the variables from a wrapper script or shell startup file.
• collector/claude-code-o11y-local-collector.yaml: the local collector overlay for local-collector mode. Configures an OTLP HTTP receiver bound to collector_receiver_endpoint when supplied, otherwise to the parsed host and port of local_collector_endpoint; a SignalFx metrics exporter (send_otlp_histograms: true), an OTLP APM traces exporter, an OTLP/HTTP logs exporter, an optional Galileo OTLP traces exporter with Galileo-API-Key, project, and logstream headers, and pipelines that fan out traces to both back ends.
• bin/claude-code-otel-headers.sh: the otelHeadersHelper shim used in splunk-direct mode and for placeholder-backed external OTLP/HTTP headers. It reads the direct token file or named runtime environment variables and writes JSON on stdout. Literal credentials never appear in settings.json.
• runtime/galileo-handoff.md: companion handoff for provisioning the Galileo project and log stream through galileo-platform-setup, including the direct REST API fallback for operators who cannot invoke that skill.
• runtime/shared-collector-routing.md: routing pattern for gateways that multiplex Codex, Claude Code, and other agents through one OTLP receiver.
• apply-plan.json, coverage-report.json, coverage-report.md, doctor-report.md, handoff.md, and metadata.json.

Galileo Integration

Galileo Observe requires a project and at least one log stream to receive traces. This skill does not create Galileo resources directly. It hands off to galileo-platform-setup for project and log-stream provisioning, then renders the collector overlay with the operator-supplied names.

The renderer does not assume a Galileo endpoint. It derives one from the user-confirmed console URL: public app.galileo.ai maps to api.galileo.ai, console. maps to api., and console- maps to api-, then /otel/traces is appended. Use --galileo-otel-endpoint for custom layouts.

Galileo authentication is a single header, Galileo-API-Key, plus routing headers project and logstream. The API key is read from GALILEO_API_KEY_FILE at collector process start; the rendered overlay references ${env:GALILEO_API_KEY}. A wrapper script sources the file into the environment immediately before invoking the collector.

Galileo trace ingest is disabled when destination is splunk-direct. There is no way to attach a second auth header to Claude Code's global OTEL_EXPORTER_OTLP_HEADERS, and re-using the same header for two back ends is unsafe.

These exported traces can ground Galileo AI Assistant beta investigations, but this skill does not enable or query the Assistant. Use galileo-platform-setup for the enterprise enablement, LLM-integration readiness, evidence-link verification, and reviewed-remediation handoff added for the July 7, 2026 Galileo release.

Provider And Model Normalization

The collector infers common Anthropic and Bedrock identities. Use explicit provider and model aliases for gateways or opaque inference-profile ARNs; see collector fan-out for normalization behavior.

Content Capture Gating

Content capture is opt-in. The following env flags are all off by default and require --accept-content-capture to render:

• OTEL_LOG_USER_PROMPTS=1: emit user prompt text in the claude_code.user_prompt log event.
• OTEL_LOG_ASSISTANT_RESPONSES=1: emit assistant reply text in claude_code.assistant_response.
• OTEL_LOG_TOOL_DETAILS=1: emit tool argument and result metadata for claude_code.tool_* events.
• OTEL_LOG_TOOL_CONTENT=1: emit tool argument and result content bodies.
• OTEL_LOG_RAW_API_BODIES=1: emit full Messages API request/response bodies through log events, or use file:/absolute/directory for local body files.

Content capture routes through Claude Code's OTLP logs exporter and, for detailed beta tracing, is also attached to span attributes (tool_input, response.model_output, etc.). Whatever back end receives the log events and traces also receives the captured content. Redact before enabling.

Version note: OTEL_LOG_ASSISTANT_RESPONSES requires Claude Code v2.1.193 or later. On those releases, an unset response flag inherits OTEL_LOG_USER_PROMPTS; the renderer therefore emits an explicit 0 for a prompt-only capture profile. Older CLIs do not provide the current assistant response log event. Applying a response-capture profile fails closed when the installed CLI is older than v2.1.193. The response flag alone does not populate Galileo: it emits an OTLP log event, while Galileo ingests traces. Detailed beta tracing plus the Galileo-only content transform are what copy the corresponding trace attributes into Galileo's Input/Output schema.

The same Galileo-only transform converts Claude's compact advertised-tool array (name plus definition hash) into one dynamic OpenInference llm.tools.<index>.tool.json_schema attribute per tool and an OTel gen_ai.tool.definitions inventory. The mapping has no fixed tool-count limit and makes Tool Selection Quality eligible for built-in and MCP tools. It emits only the observed tool name: Claude sends descriptions and parameter schemas as separate correlated log records, so the collector must not invent requirements that were not present on the LLM span.

Cardinality Flags

Session and account attributes are enabled by default and can be high-cardinality in large fleets. Review every default and the resource-attribute opt-out in reference.md.

Apply Sections

• settings: write the rendered env block into ~/.claude/settings.json (user scope) or <repo>/.claude/settings.json (project scope). The managed env keys are merged into an existing settings file; other keys are preserved. A timestamped sibling backup is created before an existing settings file is atomically replaced. A skill-generated otelHeadersHelper is reconciled across modes; an unrelated operator helper is preserved.
• env-helper: install rendered shell env helper files and, for direct or dynamic external OTLP/HTTP auth, copy bin/claude-code-otel-headers.sh into the stable otelHeadersHelper path and mark it executable.
• collector-overlay: copy the local collector overlay to an operator-owned path (defaults to reporting the render path; the operator applies it through their collector deployment workflow).
• galileo-handoff: emit a doctor entry pointing at galileo-platform-setup for project and log-stream provisioning.
• all: run every section.

--apply consumes the reviewed apply-plan.json already present in --output-dir. If no apply plan exists, the skill first regenerates the entire output directory from the current options. That render validates before it clears anything, so a render that fails validation leaves any previously good settings/, env/, collector/, bin/, and runtime/ artifacts intact — re-run --render with valid options to regenerate.

Settings Scope

--settings-scope user (default) writes ~/.claude/settings.json. --settings-scope project writes <current-repo>/.claude/settings.json. --settings-scope managed renders to <output-dir>/settings/managed-settings.json for enterprise-managed deployment; install it to the platform managed-settings path yourself (macOS /Library/Application Support/ClaudeCode/managed-settings.json, Linux/WSL /etc/claude-code/managed-settings.json, Windows C:\ProgramData\ClaudeCode\managed-settings.json), which requires elevated privileges. All scopes share the same env merge semantics.

Read reference.md for the full option contract, source basis, metric and event catalogs, and collector overlay shape.