Splunk Observability Azure Integration

Splunk Observability Cloud — Azure Integration Setup

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Connect Splunk Observability Cloud to Azure Monitor, configure the Azure integration, manage service-principal credential files, onboard multiple subscriptions, or set up Azure dashboards, detectors, logs, AKS telemetry, Log Observer.
• Preview and review the splunk observability azure integration workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-azure-integration/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-azure-integration/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Shared add-on completion gate

If this workflow installs or hands off the registry-listed Splunk Azure add-on or dashboard companion, follow the shared completion gate. Package delivery alone is not success; validate applicable ingest, macros, and shipped dashboards against data.

Render-first skill that owns the complete lifecycle of the Splunk O11y Azure integration. Rendering and rollback-plan review are network-free. Mutations require explicit --apply; --discover, --quickstart-from-live, and --validate --live are the explicit read-only modes that call the live API.

Coverage Model

Safety Rules

• Never ask for the Azure client secret (secretKey) in conversation.
• Never pass appId or secretKey as CLI arguments or env-var prefixes.
• Use --app-id-file and --secret-file (chmod 600) for file-based delivery.
• Use write_secret_file.sh to create secret files without shell-history exposure.
• Reject direct-secret flags: --secret, --client-secret, --token, --password, --app-secret.
• appId and secretKey are omitted from GET /v2/integration/<id>. The skill compares local file hashes to state/credential-hashes.json rather than server state.

Five-mode UX

Additional modes: --discover, --quickstart-from-live, --explain, --rollback, --list-services.

Primary Workflow

1. Copy and edit the spec

cp skills/splunk-observability-azure-integration/template.example my-azure-spec.yaml
# fill in realm, tenant_id, subscriptions, services

2. Write Service Principal credentials to files (once)

# Create the Azure SP and save credentials (never put secrets in history):
az ad sp create-for-rbac \
  --name splunk-observability-o11y \
  --role "Monitoring Reader" \
  --scopes "/subscriptions/${AZ_SUB_ID}" \
  --years 2 --output json > /tmp/azure-sp.json && chmod 600 /tmp/azure-sp.json

# Extract and write app ID (not a secret, but keep consistent):
jq -r .appId /tmp/azure-sp.json > /tmp/azure-app-id.txt && chmod 600 /tmp/azure-app-id.txt

# Write the client secret to a separate file:
jq -r .password /tmp/azure-sp.json > /tmp/azure-secret.txt && chmod 600 /tmp/azure-secret.txt

rm /tmp/azure-sp.json

3. Render

bash skills/splunk-observability-azure-integration/scripts/setup.sh \
  --render \
  --spec my-azure-spec.yaml \
  --realm us1

4. Review the plan

splunk-observability-azure-integration-rendered/
  01-overview.md          # plan summary + apply command
  02-services.md          # services subscription plan
  03-auth.md              # SP auth plan
  04-validation.md        # validation steps
  rest/create.json        # POST /v2/integration body
  rest/update.json        # PUT /v2/integration/{id} body
  azure-cli/create-sp.sh  # az ad sp create-for-rbac (review)
  azure-cli/grant-reader.sh # role assignment
  bicep/role-assignment.bicep # Bicep subscription-scope role assignment
  terraform/main.tf       # signalfx_azure_integration resource
  terraform/variables.tf  # variable declarations
  handoffs/               # cross-skill handoff drivers
  coverage-report.json    # per-section coverage status

5. Apply

bash skills/splunk-observability-azure-integration/scripts/setup.sh \
  --apply \
  --spec my-azure-spec.yaml \
  --realm us1 \
  --token-file /tmp/splunk_o11y_token \
  --app-id-file /tmp/azure-app-id.txt \
  --secret-file /tmp/azure-secret.txt

Quickstart

bash skills/splunk-observability-azure-integration/scripts/setup.sh \
  --quickstart \
  --spec my-azure-spec.yaml \
  --realm us1

Doctor

bash skills/splunk-observability-azure-integration/scripts/setup.sh \
  --doctor \
  --realm us1

Doctor checks: services non-empty, poll-rate 60–600, namedToken ForceNew warning, AZURE_US_GOVERNMENT + non-GovCloud realm mismatch, credential-hash freshness, and the appId/secretKey GET-omission/reinjection notice.

Rollback

Rollback is a three-step snapshot, offline review, and exact-plan apply workflow. It never resolves a mutation target by name.

1. Capture a read-only observed snapshot

bash skills/splunk-observability-azure-integration/scripts/setup.sh \
  --discover \
  --realm us1 \
  --token-file /secure/splunk_o11y_token \
  --output-dir azure-live

2. Render and review a disable plan offline

bash skills/splunk-observability-azure-integration/scripts/setup.sh \
  --rollback disable \
  --realm us1 \
  --integration-id SERVER_ASSIGNED_ID \
  --integration-name EXACT_NAME \
  --observed-state-file azure-live/state/current-state.json \
  --app-id-file /secure/azure-app-id \
  --secret-file /secure/azure-secret \
  --plan-file azure-live/state/disable-plan.json

The renderer reads the local credential files only to bind their SHA-256 digests; the plan contains no credential values or paths and makes no network request. Review the exact JSON and printed hash.

3. Apply that exact reviewed plan

bash skills/splunk-observability-azure-integration/scripts/setup.sh \
  --rollback disable --apply \
  --realm us1 \
  --integration-id SERVER_ASSIGNED_ID \
  --plan-file azure-live/state/disable-plan.json \
  --plan-hash REVIEWED_SHA256 \
  --accept-disable-integration SERVER_ASSIGNED_ID \
  --token-file /secure/splunk_o11y_admin_token \
  --app-id-file /secure/azure-app-id \
  --secret-file /secure/azure-secret

Disable is only an enabled-to-disabled transition and reconstructs the write-only appId and secretKey fields from the same reviewed files. Delete requires a separately rendered --rollback delete plan and --accept-delete-integration SERVER_ASSIGNED_ID; it rejects all Azure credential-file flags. Never use delete as a workaround when disable is blocked. Bare --rollback renders a disable plan but cannot be applied, and --rollback integration remains only as a deprecated disable alias.

See reference.md for plan schema, locking, replay, reconciliation, and remote-race behavior.

Hand-offs

• Logs path → splunk-app-install for Splunk_TA_microsoft-cloudservices (Splunkbase 3110)
• AKS host telemetry → splunk-observability-otel-collector-setup
• Azure dashboards → splunk-observability-dashboard-builder
• Azure detectors → splunk-observability-native-ops
• Log Observer Connect → splunk-observability-cloud-integration-setup
• HEC tokens → splunk-hec-service-setup
• AppDynamics on Azure → splunk-appdynamics-setup

Out of Scope

• Azure Event Hub streaming (not in the Splunk O11y wire contract as a separate integration type; importAzureMonitor controls metric-vs-metadata-only)
• Azure log ingestion (Splunk_TA_microsoft-cloudservices, Splunkbase 3110 — handed off)
• Managed Identity authentication (Splunk O11y requires a Service Principal)
• Workload Identity Federation for Azure (not in the wire contract)
• Azure China (AzureChinaCloud) — not supported by this integration

Validation

bash skills/splunk-observability-azure-integration/scripts/validate.sh \
  --output-dir splunk-observability-azure-integration-rendered

Static checks: required files, JSON shape (type: Azure), no secret-looking content in rendered files. With --live: GET /v2/integration probe.