Splunk Observability GCP Integration

Splunk Observability Cloud — GCP Integration Setup

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Connect Splunk Observability Cloud to GCP metrics, configure Service Account or official generated WIF credentials, manage the GCP REST integration, or set up GCP dashboards, detectors, logs, or GKE telemetry handoffs.
• Preview and review the splunk observability gcp integration workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-gcp-integration/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-gcp-integration/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Shared add-on completion gate

If this workflow installs or hands off the registry-listed Splunk GCP add-on or dashboard companion, follow the shared completion gate. Package delivery alone is not success; validate applicable ingest, macros, and shipped dashboards against data.

Render-first skill that owns the complete lifecycle of the Splunk O11y GCP integration. Rendering and rollback-plan review are network-free. Mutations require explicit --apply; --discover, --quickstart-from-live, and --validate --live are the explicit read-only modes that call the live API.

Splunk Platform Add-on Verification Boundary

The separate log-ingestion handoff uses Splunkbase 3088. Its package-derived baseline is now 5.1.1, which is also the current public release and advertises Splunk 10.5 support; that package was downloaded, unpacked, and inspected here, so the shared installer's default pin needs no review override. This does not change the Observability Cloud GCP metrics REST integration owned by this skill.

Coverage Model

Safety Rules

• Never ask for the GCP Service Account JSON key (projectKey) in conversation.
• Never pass projectKey as a CLI argument or env-var prefix.
• Use --key-file (chmod 600) for file-based delivery. Each JSON file is securely parsed and mapped by its own project_id, independent of flag order. Missing, extra, duplicate, or mismatched project coverage fails before any live request; a credential is never reused for another project.
• In WIF mode, use only Splunk's official generated file named gcp_wif_config.json, stored unchanged as a regular mode-600 file. Pass its path through --wif-config-file; never paste its contents into the spec.
• Do not infer realm principals or construct WIF pool/provider values. The generated document is opaque and is sent as compact, stringified JSON in workloadIdentityFederationConfig.
• Use write_secret_file.sh to create secret files without shell-history exposure.
• Reject direct-secret flags: --secret, --password, --api-key, --project-key, --token, --wif-config.
• projectKey is not returned by GET /v2/integration/<id>. The skill compares local file hashes to state/credential-hashes.json rather than server state.
• Newly rendered provisioning payloads include projects.syncMode; SELECTED also includes the reviewed project ID list. Rollback preserves a valid service-account live object that omits projects, while WIF requires a validated projects contract.

Five-mode UX

Additional modes: --discover, --quickstart-from-live, --explain, --rollback, --list-services.

Primary Workflow

1. Copy and edit the spec

cp skills/splunk-observability-gcp-integration/template.example my-gcp-spec.yaml
# fill in realm, project_id, services

2. Write Service Account key to a file (once)

# Download the SA key from GCP Console or CLI (never put in history):
gcloud iam service-accounts keys create /tmp/gcp-sa-key.json \
  --iam-account=splunk-o11y@${GCP_PROJECT_ID}.iam.gserviceaccount.com
chmod 600 /tmp/gcp-sa-key.json

3. Render

bash skills/splunk-observability-gcp-integration/scripts/setup.sh \
  --render \
  --spec my-gcp-spec.yaml \
  --realm us1

4. Review the plan

splunk-observability-gcp-integration-rendered/
  README.md               # plan summary + apply command
  rest/create.json        # POST /v2/integration body
  rest/update.json        # PUT /v2/integration/{id} body
  rest/wif-config-file-manifest.json # file path only; no WIF contents
  gcloud-cli/create-sa.sh  # gcloud iam sa create (review)
  gcloud-cli/bind-roles.sh # role bindings
  terraform/main.tf       # SA-key resource; WIF intentionally uses the REST path
  terraform/variables.tf  # variable declarations
  handoffs/               # cross-skill handoff drivers
  state/                  # populated on apply
  coverage-report.json    # per-section coverage status
  apply-plan.json         # ordered steps

5. Apply

bash skills/splunk-observability-gcp-integration/scripts/setup.sh \
  --apply \
  --spec my-gcp-spec.yaml \
  --realm us1 \
  --token-file /tmp/splunk_o11y_token \
  --key-file /tmp/gcp-sa-key.json

Quickstart

bash skills/splunk-observability-gcp-integration/scripts/setup.sh \
  --quickstart \
  --spec my-gcp-spec.yaml \
  --realm us1

Doctor

bash skills/splunk-observability-gcp-integration/scripts/setup.sh \
  --doctor \
  --realm us1

Doctor checks: services non-empty when explicit mode, poll-rate 60–600, projects.syncMode, namedToken ForceNew warning, credential-hash freshness, and WIF file existence, JSON integrity, filename, regular-file status, and mode.

Rollback

Rollback is a three-step snapshot, offline review, and exact-plan apply workflow. It never resolves a mutation target by name.

1. Capture a read-only observed snapshot

bash skills/splunk-observability-gcp-integration/scripts/setup.sh \
  --discover \
  --realm us1 \
  --token-file /secure/splunk_o11y_token \
  --output-dir gcp-live

2. Render and review a disable plan offline

bash skills/splunk-observability-gcp-integration/scripts/setup.sh \
  --rollback disable \
  --realm us1 \
  --integration-id SERVER_ASSIGNED_ID \
  --integration-name EXACT_NAME \
  --observed-state-file gcp-live/state/current-state.json \
  --key-file /secure/project-a.json \
  --key-file /secure/project-b.json \
  --plan-file gcp-live/state/disable-plan.json

For WIF, replace every --key-file with one --wif-config-file /secure/gcp_wif_config.json. The renderer reads credentials only to validate them and bind secret-free SHA-256 metadata; it makes no network request. Review the exact JSON and printed hash.

3. Apply that exact reviewed plan

bash skills/splunk-observability-gcp-integration/scripts/setup.sh \
  --rollback disable --apply \
  --realm us1 \
  --integration-id SERVER_ASSIGNED_ID \
  --plan-file gcp-live/state/disable-plan.json \
  --plan-hash REVIEWED_SHA256 \
  --accept-disable-integration SERVER_ASSIGNED_ID \
  --token-file /secure/splunk_o11y_admin_token \
  --key-file /secure/project-a.json \
  --key-file /secure/project-b.json

Disable is only an enabled-to-disabled transition and reconstructs the live authentication shape from the reviewed credential mode. Delete requires a separately rendered --rollback delete plan and --accept-delete-integration SERVER_ASSIGNED_ID; it rejects all key and WIF files. Never use delete as a workaround when disable is blocked. Bare --rollback renders a disable plan but cannot be applied, and --rollback integration remains only as a deprecated disable alias.

See reference.md for plan schema, locking, replay, reconciliation, compatibility, and remote-race behavior.

Workload Identity Federation

WIF avoids a Service Account private key. Obtain gcp_wif_config.json from Splunk's supported integration workflow and keep it unchanged. The skill does not generate that document and does not infer any realm identity.

authentication:
  mode: workload_identity_federation
  workload_identity_federation:
    config_file: "/secure/path/gcp_wif_config.json"

chmod 600 /secure/path/gcp_wif_config.json
bash skills/splunk-observability-gcp-integration/scripts/setup.sh \
  --apply --spec my-wif-spec.yaml --realm us1 \
  --token-file /secure/path/splunk_o11y_token \
  --wif-config-file /secure/path/gcp_wif_config.json

The live request uses authMethod: WORKLOAD_IDENTITY_FEDERATION and injects the complete document as compact/stringified JSON in workloadIdentityFederationConfig. The upstream SignalFx Terraform provider supports WIF and projects arguments. This skill intentionally keeps WIF on its reviewed REST path and does not currently generate the Terraform WIF resource or gcloud pool/provider scripts.

Hand-offs

• Logs path → splunk-app-install for Splunk_TA_google_cloud (Splunkbase 3088)
• GKE host telemetry → splunk-observability-otel-collector-setup
• GCP dashboards → splunk-observability-dashboard-builder
• GCP detectors → splunk-observability-native-ops
• Log Observer Connect → splunk-observability-cloud-integration-setup
• HEC tokens → splunk-hec-service-setup

Out of Scope

• GCP Pub/Sub streaming (not in the Splunk O11y wire contract)
• GCP log ingestion (Splunk_TA_google_cloud, Splunkbase 3088 — handed off)
• GCP China regions — not supported by this integration

Validation

bash skills/splunk-observability-gcp-integration/scripts/validate.sh \
  --output-dir splunk-observability-gcp-integration-rendered

Static checks: required files, JSON shape (type: GCP), no secret-looking content in rendered files. With --live: GET /v2/integration probe.