Splunk Observability AWS Integration

Splunk Observability Cloud <-> AWS Integration

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Connect AWS to Splunk Observability Cloud, configure CloudWatch Metric Streams, render IAM policies, manage the AWSCloudWatch REST/Terraform object, monitor Bedrock metrics, set up multi-account AWS Organizations onboarding, audit drift,.
• Preview and review the splunk observability aws integration workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-aws-integration/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-aws-integration/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Render-first skill that owns the entire AWSCloudWatch integration object in Splunk Observability Cloud. The skill is standalone reusable and does not require any other Splunk Observability skill to run, but it hands off to other skills for adjacent surfaces (logs, Lambda APM, dashboards, detectors, OTel collectors).

The workflow is render-first by default. Live API changes only happen when the user explicitly asks for --apply.

Coverage Model

Every rendered section gets an explicit coverage status in coverage-report.json:

• api_apply — a documented public REST API supports create / update / delete / validate (e.g. POST /v2/integration).
• api_validate — a documented public REST API supports read or validation only (e.g. preflighting ec2:DescribeRegions permission via the live integration's reported region list).
• deeplink — the skill renders a deterministic Splunk Observability Cloud UI link and validates referenced data where an API allows (e.g. the AWS guided-setup deeplink for AWS-managed Metric Streams).
• handoff — the skill renders deterministic operator steps for cross-skill workflows (e.g. logs path -> splunk-app-install for Splunk_TA_AWS).
• not_applicable — the section does not apply to the chosen target (e.g. Splunk-managed Metric Streams permissions when connection.mode is polling, or any AWS integration on realm: us2-gcp because that realm is GCP-hosted and has no AWS STS region mapping).

Safety Rules

• Never ask for Splunk Observability tokens, AWS access keys, or AWS secret access keys in conversation.
• Never pass any secret on the command line or as an environment-variable prefix.
• Use --token-file for the Splunk Observability Cloud admin user API access token. Token files must be chmod 600. Override only with --allow-loose-token-perms (emits a WARN — use only for short-lived scratch tokens).
• Secret files must always be regular, non-symlink, single-hardlink files no larger than 64 KiB containing one non-empty printable-ASCII value with at most one trailing LF or CRLF. Readers use O_NOFOLLOW, bounded double reads, and stable metadata/content fingerprints; --allow-loose-token-perms relaxes only the mode check.
• Authenticated API calls are pinned to supported HTTPS realm hosts and refuse every redirect so X-SF-Token cannot be forwarded to another URL.
• For authentication.mode: security_token (GovCloud / China), use --aws-access-key-id-file and --aws-secret-access-key-file. Both files must be chmod 600.
• Reject direct secret flags: --token, --access-token, --api-token, --o11y-token, --admin-token, --sf-token, --external-id, --aws-access-key-id, --aws-secret-access-key, --aws-secret-key, --password.
• Prefer SPLUNK_O11Y_REALM and SPLUNK_O11Y_TOKEN_FILE from the repo credentials file when present; these store only realms and token-file paths, never token values.
• Strip every secret from 00-09-*.md, apply-plan.json, payloads/, current-state.json, state/apply-state.json, and any other rendered artifact on disk. The Splunk Observability admin token is referenced as ${SPLUNK_O11Y_TOKEN_FILE} everywhere, never inlined.
• The renderer FAILs render when the operator passes the deprecated enableLogsSync field with a clear pointer to the Splunk_TA_AWS handoff.
• The renderer FAILs render when regions: [] (the canonical schema rejects an empty list and Splunk highly discourages it because new AWS regions auto-onboard and inflate cost).
• bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_o11y_token helps the user create a token file without exposing the secret in shell history.

Five-mode UX

Plus quality-of-life flags:

• --quickstart-from-live — turn the live integration into a template.observed.yaml the operator can edit; never overwrites their template.example.
• --explain — print the apply plan in plain English, no API calls; for change-management approvals.
• --rollback <section> — render (do not auto-run) the reverse-engineered commands for steps that have a public reversible API.
• --list-namespaces — print the supported AWS service / namespace catalog (mirrors references/namespaces-catalog.md).
• --list-recommended-stats — print the per-namespace per-metric stat catalog used by collect_only_recommended_stats: true.
• --privatelink-domain {legacy,new} — pick signalfx.com (default; matches current Splunk-published PrivateLink doc) vs observability.splunkcloud.com PrivateLink hostnames.
• --cfn-template-url URL — override the default https://o11y-public.s3.amazonaws.com/aws-cloudformation-templates/release/template_metric_streams_regional.yaml (or the StackSets equivalent when metric_streams.use_stack_sets: true). Overrides must be absolute HTTPS URLs on port 443, end in .yaml or .yml, and contain no credentials, query, fragment, whitespace, or control characters. The rendered AWS CLI command shell-quotes the validated URL.
• --accept-drift FIELD[,FIELD...] — needed for --apply when discover shows the live integration differs from the rendered spec on a field with side effects (e.g. flipping useMetricStreamsSync).

Primary Workflow

1. Collect non-secret values: realm, integration name, AWS account ID(s), IAM role name, regions, services list (or all_built_in), connection mode (polling vs Splunk-managed Metric Streams vs AWS-managed Metric Streams), custom namespaces, multi-account toggle.

2. Create or update a JSON / YAML spec from template.example.

3. Render and validate:

   bash skills/splunk-observability-aws-integration/scripts/setup.sh \
     --render \
     --spec skills/splunk-observability-aws-integration/template.example \
     --output-dir splunk-observability-aws-integration-rendered
   

4. Review splunk-observability-aws-integration-rendered/:

   • README.md — TL;DR and ordered next-step commands.
   • architecture.mmd — Mermaid topology of the rendered integration.
   • 00-prerequisites.md through 09-handoff.md — numbered per-section plans.
   • coverage-report.json — per-section coverage status.
   • apply-plan.json — apply ordering with idempotency keys (no secrets).
   • payloads/ — per-step request bodies for REST calls and CFN parameters.
   • aws/ — CloudFormation template stubs (regional or StackSets) and Terraform .tf files for the AWS-side resources and the Splunk-side integration object.
   • iam/ — per-use-case IAM JSON (foundation, polling, streams, tag-sync, Cassandra-special-case, GovCloud security-token).
   • scripts/ — per-step apply scripts and cross-skill handoff drivers.
   • support-tickets/ — pre-filled tickets when Splunk Support is required.

5. Apply SecurityToken mode only when explicitly requested. ExternalId mode uses the rendered two-phase handoff and direct apply refuses a partial disabled integration:

   The rendered iam/iam-trust.json intentionally uses ${SPLUNK_AWS_ACCOUNT_ID_FROM_POST_RESPONSE} until the create response is available. After POST /v2/integration, replace that placeholder with the returned sfxAwsAccountArn before deploying the IAM trust policy.

   bash skills/splunk-observability-aws-integration/scripts/setup.sh \
     --apply integration,validation \
     --spec skills/splunk-observability-aws-integration/template.example \
     --realm us1 \
     --token-file /tmp/splunk_o11y_admin_token \
     --aws-access-key-id-file /tmp/aws_access_key_id \
     --aws-secret-access-key-file /tmp/aws_secret_access_key
   

   To run only a subset of sections:

   bash skills/splunk-observability-aws-integration/scripts/setup.sh \
     --apply integration \
     --spec my-security-token-aws-integration.yaml \
     --token-file /tmp/splunk_o11y_admin_token \
     --aws-access-key-id-file /tmp/aws_access_key_id \
     --aws-secret-access-key-file /tmp/aws_secret_access_key
   

Supported Sections

Specs use api_version: splunk-observability-aws-integration/v1 and can include the following top-level blocks (full reference in reference.md):

• prerequisites — region/realm preflight, FedRAMP/GovCloud/GCP carve-out, IAM permission check stubs, CFN template URL HTTP HEAD probe.
• authentication — external_id or security_token mode, AWS account ID, IAM role name, returned external ID.
• connection — polling vs Splunk-managed vs AWS-managed vs Terraform-only mode; pollRate, metadataPollRate, inactiveMetricsPollRate.
• regions — explicit AWS region list (cannot be empty).
• services — all_built_in / explicit / namespace_filtered / custom_only; collect_only_recommended_stats; metric_stats_to_syncs; namespace_sync_rules (for built-in AWS/* namespaces).
• custom_namespaces — simple_list (serializes to customCloudwatchNamespaces) OR sync_rules (serializes to customNamespaceSyncRules); they conflict. sync_custom_namespaces_only toggle.
• guards — enable_check_large_volume, ignore_all_status_metrics, sync_load_balancer_target_group_tags, enable_aws_usage.
• metric_streams — use_metric_streams_sync, managed_externally, named_token, cloudformation, cloudformation_template_url, use_stack_sets, terraform.
• private_link — enable, endpoint_types, service_name_overrides.
• terraform_provider — source (splunk-terraform/signalfx), version (~> 9.0 default).
• multi_account — enabled, control_account_id, member_accounts, cfn_stacksets.
• handoffs — lambda_apm, logs_via_splunk_ta_aws, dashboards, detectors, otel_collector_for_ec2_eks.

Hand-offs to Other Skills

• AWS log ingestion -> bash skills/splunk-app-install/scripts/install_app.sh --source splunkbase --app-id 1876 (Splunk Add-on for AWS, Splunkbase 1876, min v8.1.1). Renderer preflights for Splunk_TA_amazon_security_lake and emits an uninstall step before the v7+ upgrade.
• Log Observer Connect to surface those logs in O11y -> splunk-observability-cloud-integration-setup.
• Native O11y dashboards for AWS namespaces -> splunk-observability-dashboard-builder.
• Detectors / AutoDetect alerting -> splunk-observability-native-ops.
• OTel collector on EC2 / EKS for richer host telemetry than CWAgent -> splunk-observability-otel-collector-setup.
• Lambda APM via the Splunk OpenTelemetry Lambda layer (publisher 254067382080) -> splunk-observability-aws-lambda-apm-setup. Renderer emits a hand-off stub in 09-handoff.md.

Out of Scope

• AWS log collection via the Splunk AWS log collector Lambda (handed off to Splunk_TA_AWS; the enableLogsSync API field is deprecated and rejected by the renderer).
• Lambda APM instrumentation via the OpenTelemetry Lambda layer (handed off to splunk-observability-aws-lambda-apm-setup).
• Native O11y AWS dashboard / detector CRUD (handed off to splunk-observability-dashboard-builder and splunk-observability-native-ops).
• AppDynamics for AWS workloads (separate AppDynamics SaaS workflow).
• Splunk Cloud Platform Data Manager AWS log onboarding (different product; Splunk Cloud Platform side, not Splunk Observability Cloud).

Validation

bash skills/splunk-observability-aws-integration/scripts/validate.sh \
  --output-dir splunk-observability-aws-integration-rendered

Static checks: required-files, IAM JSON shape, and a secrets-leak scan across every rendered file.

With --live, the CFN HTTPS probe is mandatory only when the rendered plan uses that template. The live API check requires exactly one enabled integration matching the rendered name, AWS account, regions, polling/stream mode, services, custom namespaces, rates, and guards, then calls GET /v2/integration/validate/{id}. Missing credentials, tools, or evidence fail validation. The validation endpoint requires a short-lived administrator User API session token.

For the cross-domain AWS/EKS/O11y production gate, render clean packets with scripts/staging/render-aws-eks-o11y.py and run scripts/staging/validate-aws-eks-o11y.sh. See ../../scripts/staging/README.md. The gate is read-only and records a private, sanitized JSON report.

See reference.md and the focused references under references/ for full detail. The plan and corrections that produced this skill are recorded in .cursor/plans/splunk_o11y_aws_integration_skill_*.plan.md.