Splunk Observability Cloud <-> AWS Integration
Prerequisites
| Tool or access |
Purpose |
Verify |
| Bash and Python 3 |
Run bundled setup and validation helpers |
bash --version && python3 --version |
| Required product/platform access |
Inspect or configure the selected target |
Complete the documented preflight |
| Credential files for live modes |
Keep secrets out of chat |
Verify paths only |
Workflow Overview
┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘
When to Activate
- Connect AWS to Splunk Observability Cloud, configure CloudWatch Metric Streams, render IAM policies, manage the
AWSCloudWatch REST/Terraform object, monitor Bedrock metrics, set up multi-account AWS Organizations onboarding,
audit drift,.
- Preview and review the splunk observability aws integration workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.
Scope
Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.
Examples
Inspect the supported setup modes before selecting one:
bash skills/splunk-observability-aws-integration/scripts/setup.sh --help
Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.
Inspect validation modes before running completion checks:
bash skills/splunk-observability-aws-integration/scripts/validate.sh --help
Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.
Troubleshooting
| Issue |
Cause |
Resolution |
| Preflight fails |
A required tool or access path is missing |
Resolve it before rendering or applying |
| Rendered assets are incomplete |
Required non-secret inputs are absent |
Complete intake and render again |
| Apply is blocked |
Review, credentials, or explicit acceptance is missing |
Use the documented handoff |
| Validation is incomplete |
Live evidence is unavailable |
Record the gap and keep completion open |
Render-first skill that owns the entire AWSCloudWatch integration object in
Splunk Observability Cloud. The skill is standalone reusable and does not
require any other Splunk Observability skill to run, but it hands off to other
skills for adjacent surfaces (logs, Lambda APM, dashboards, detectors, OTel
collectors).
The workflow is render-first by default. Live API changes only happen when the
user explicitly asks for --apply.
Coverage Model
Every rendered section gets an explicit coverage status in
coverage-report.json:
api_apply — a documented public REST API supports create / update / delete /
validate (e.g. POST /v2/integration).
api_validate — a documented public REST API supports read or validation only
(e.g. preflighting ec2:DescribeRegions permission via the live integration's
reported region list).
deeplink — the skill renders a deterministic Splunk Observability Cloud UI
link and validates referenced data where an API allows (e.g. the AWS
guided-setup deeplink for AWS-managed Metric Streams).
handoff — the skill renders deterministic operator steps for cross-skill
workflows (e.g. logs path -> splunk-app-install for Splunk_TA_AWS).
not_applicable — the section does not apply to the chosen target (e.g.
Splunk-managed Metric Streams permissions when connection.mode is polling,
or any AWS integration on realm: us2-gcp because that realm is GCP-hosted
and has no AWS STS region mapping).
Safety Rules
- Never ask for Splunk Observability tokens, AWS access keys, or AWS secret
access keys in conversation.
- Never pass any secret on the command line or as an environment-variable
prefix.
- Use
--token-file for the Splunk Observability Cloud admin user API access
token. Token files must be chmod 600. Override only with
--allow-loose-token-perms (emits a WARN — use only for short-lived scratch
tokens).
- Secret files must always be regular, non-symlink, single-hardlink files no
larger than 64 KiB containing one non-empty printable-ASCII value with at
most one trailing LF or CRLF. Readers use
O_NOFOLLOW,
bounded double reads, and stable metadata/content fingerprints;
--allow-loose-token-perms relaxes only the mode check.
- Authenticated API calls are pinned to supported HTTPS realm hosts and refuse
every redirect so
X-SF-Token cannot be forwarded to another URL.
- For
authentication.mode: security_token (GovCloud / China), use
--aws-access-key-id-file and --aws-secret-access-key-file. Both files
must be chmod 600.
- Reject direct secret flags:
--token, --access-token, --api-token,
--o11y-token, --admin-token, --sf-token, --external-id,
--aws-access-key-id, --aws-secret-access-key, --aws-secret-key,
--password.
- Prefer
SPLUNK_O11Y_REALM and SPLUNK_O11Y_TOKEN_FILE from the repo
credentials file when present; these store only realms and token-file
paths, never token values.
- Strip every secret from
00-09-*.md, apply-plan.json, payloads/,
current-state.json, state/apply-state.json, and any other rendered
artifact on disk. The Splunk Observability admin token is referenced as
${SPLUNK_O11Y_TOKEN_FILE} everywhere, never inlined.
- The renderer FAILs render when the operator passes the deprecated
enableLogsSync field with a clear pointer to the Splunk_TA_AWS handoff.
- The renderer FAILs render when
regions: [] (the canonical schema rejects an
empty list and Splunk highly discourages it because new AWS regions
auto-onboard and inflate cost).
bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_o11y_token
helps the user create a token file without exposing the secret in shell
history.
Five-mode UX
| Mode |
Flag |
Purpose |
| quickstart |
--quickstart |
Render the common scenario and print the supported SecurityToken apply plus ExternalId handoff. No live changes. |
| render |
--render (default) |
Produces the numbered plan tree under --output-dir. Never touches live state. |
| discover |
--discover |
Read-only sweep that polls GET /v2/integration?type=AWSCloudWatch, writes current-state.json, and emits a drift-report.md against the rendered plan. |
| doctor |
--doctor |
Runs the troubleshooting catalog and emits doctor-report.md with prioritized fixes and the exact setup.sh --apply command for each fix. |
| apply |
--apply [SECTIONS] |
Applies selected sections. SecurityToken integration and regional streams can mutate; IAM, ExternalId two-phase setup, and StackSets fail closed to explicit handoffs. |
Plus quality-of-life flags:
--quickstart-from-live — turn the live integration into a template.observed.yaml
the operator can edit; never overwrites their template.example.
--explain — print the apply plan in plain English, no API calls; for
change-management approvals.
--rollback <section> — render (do not auto-run) the reverse-engineered
commands for steps that have a public reversible API.
--list-namespaces — print the supported AWS service / namespace catalog
(mirrors references/namespaces-catalog.md).
--list-recommended-stats — print the per-namespace per-metric stat catalog
used by collect_only_recommended_stats: true.
--privatelink-domain {legacy,new} — pick signalfx.com (default; matches
current Splunk-published PrivateLink doc) vs observability.splunkcloud.com
PrivateLink hostnames.
--cfn-template-url URL — override the default
https://o11y-public.s3.amazonaws.com/aws-cloudformation-templates/release/template_metric_streams_regional.yaml
(or the StackSets equivalent when metric_streams.use_stack_sets: true).
Overrides must be absolute HTTPS URLs on port 443, end in .yaml or .yml,
and contain no credentials, query, fragment, whitespace, or control
characters. The rendered AWS CLI command shell-quotes the validated URL.
--accept-drift FIELD[,FIELD...] — needed for --apply when discover shows
the live integration differs from the rendered spec on a field with side
effects (e.g. flipping useMetricStreamsSync).
Primary Workflow
Collect non-secret values: realm, integration name, AWS account ID(s),
IAM role name, regions, services list (or all_built_in), connection
mode (polling vs Splunk-managed Metric Streams vs AWS-managed Metric
Streams), custom namespaces, multi-account toggle.
Create or update a JSON / YAML spec from template.example.
Render and validate:
bash skills/splunk-observability-aws-integration/scripts/setup.sh \
--render \
--spec skills/splunk-observability-aws-integration/template.example \
--output-dir splunk-observability-aws-integration-rendered
Review splunk-observability-aws-integration-rendered/:
README.md — TL;DR and ordered next-step commands.
architecture.mmd — Mermaid topology of the rendered integration.
00-prerequisites.md through 09-handoff.md — numbered per-section plans.
coverage-report.json — per-section coverage status.
apply-plan.json — apply ordering with idempotency keys (no secrets).
payloads/ — per-step request bodies for REST calls and CFN parameters.
aws/ — CloudFormation template stubs (regional or StackSets) and
Terraform .tf files for the AWS-side resources and the Splunk-side
integration object.
iam/ — per-use-case IAM JSON (foundation, polling, streams, tag-sync,
Cassandra-special-case, GovCloud security-token).
scripts/ — per-step apply scripts and cross-skill handoff drivers.
support-tickets/ — pre-filled tickets when Splunk Support is required.
Apply SecurityToken mode only when explicitly requested. ExternalId mode
uses the rendered two-phase handoff and direct apply refuses a partial
disabled integration:
The rendered iam/iam-trust.json intentionally uses
${SPLUNK_AWS_ACCOUNT_ID_FROM_POST_RESPONSE} until the create response is
available. After POST /v2/integration, replace that placeholder with the
returned sfxAwsAccountArn before deploying the IAM trust policy.
bash skills/splunk-observability-aws-integration/scripts/setup.sh \
--apply integration,validation \
--spec skills/splunk-observability-aws-integration/template.example \
--realm us1 \
--token-file /tmp/splunk_o11y_admin_token \
--aws-access-key-id-file /tmp/aws_access_key_id \
--aws-secret-access-key-file /tmp/aws_secret_access_key
To run only a subset of sections:
bash skills/splunk-observability-aws-integration/scripts/setup.sh \
--apply integration \
--spec my-security-token-aws-integration.yaml \
--token-file /tmp/splunk_o11y_admin_token \
--aws-access-key-id-file /tmp/aws_access_key_id \
--aws-secret-access-key-file /tmp/aws_secret_access_key
Supported Sections
Specs use api_version: splunk-observability-aws-integration/v1 and can
include the following top-level blocks (full reference in
reference.md):
prerequisites — region/realm preflight, FedRAMP/GovCloud/GCP carve-out,
IAM permission check stubs, CFN template URL HTTP HEAD probe.
authentication — external_id or security_token mode, AWS account ID,
IAM role name, returned external ID.
connection — polling vs Splunk-managed vs AWS-managed vs Terraform-only
mode; pollRate, metadataPollRate, inactiveMetricsPollRate.
regions — explicit AWS region list (cannot be empty).
services — all_built_in / explicit / namespace_filtered /
custom_only; collect_only_recommended_stats; metric_stats_to_syncs;
namespace_sync_rules (for built-in AWS/* namespaces).
custom_namespaces — simple_list (serializes to customCloudwatchNamespaces)
OR sync_rules (serializes to customNamespaceSyncRules); they conflict.
sync_custom_namespaces_only toggle.
guards — enable_check_large_volume, ignore_all_status_metrics,
sync_load_balancer_target_group_tags, enable_aws_usage.
metric_streams — use_metric_streams_sync, managed_externally,
named_token, cloudformation, cloudformation_template_url,
use_stack_sets, terraform.
private_link — enable, endpoint_types, service_name_overrides.
terraform_provider — source (splunk-terraform/signalfx), version
(~> 9.0 default).
multi_account — enabled, control_account_id, member_accounts,
cfn_stacksets.
handoffs — lambda_apm, logs_via_splunk_ta_aws, dashboards,
detectors, otel_collector_for_ec2_eks.
Hand-offs to Other Skills
Out of Scope
- AWS log collection via the Splunk AWS log collector Lambda (handed off to
Splunk_TA_AWS; the enableLogsSync API field is deprecated and rejected
by the renderer).
- Lambda APM instrumentation via the OpenTelemetry Lambda layer (handed off to
splunk-observability-aws-lambda-apm-setup).
- Native O11y AWS dashboard / detector CRUD (handed off to
splunk-observability-dashboard-builder and
splunk-observability-native-ops).
- AppDynamics for AWS workloads (separate AppDynamics SaaS workflow).
- Splunk Cloud Platform Data Manager AWS log onboarding (different product;
Splunk Cloud Platform side, not Splunk Observability Cloud).
Validation
bash skills/splunk-observability-aws-integration/scripts/validate.sh \
--output-dir splunk-observability-aws-integration-rendered
Static checks: required-files, IAM JSON shape, and a secrets-leak scan across
every rendered file.
With --live, the CFN HTTPS probe is mandatory only when the rendered plan
uses that template. The live API check requires exactly one enabled integration
matching the rendered name, AWS account, regions, polling/stream mode, services,
custom namespaces, rates, and guards, then calls
GET /v2/integration/validate/{id}. Missing credentials, tools, or evidence
fail validation. The validation endpoint requires a short-lived administrator
User API session token.
For the cross-domain AWS/EKS/O11y production gate, render clean packets with
scripts/staging/render-aws-eks-o11y.py and run
scripts/staging/validate-aws-eks-o11y.sh. See
../../scripts/staging/README.md. The gate
is read-only and records a private, sanitized JSON report.
See reference.md and the focused references under references/ for full
detail. The plan and corrections that produced this skill are recorded in
.cursor/plans/splunk_o11y_aws_integration_skill_*.plan.md.
1---2name: splunk-observability-aws-integration3description: "Use when the user asks to connect AWS to Splunk Observability Cloud, configure CloudWatch Metric4 Streams, render IAM policies, manage the AWSCloudWatch REST/Terraform object, monitor Bedrock metrics,5 set up multi-account AWS Organizations onboarding, audit drift, or migrate polling to Metric Streams.6 Hand off Lambda APM, AWS logs, dashboards, detectors, and EC2/EKS host telemetry to their owning skills.7 Render, preflight, apply, validate, discover, and diagnose the Splunk Observability Cloud AWSCloudWatch8 integration across polling, Splunk-managed Metric Streams, AWS-managed Metric Streams, and Terraform9 paths. Covers IAM trust and policy stubs, External ID and SecurityToken auth, CloudFormation and10 StackSets assets, Terraform payloads, field conflict checks, recommended stats, namespace sync rules,11 PrivateLink ingest stubs, drift adoption, and troubleshooting."12compatibility: "No direct Splunk Platform runtime dependency. This workflow can be used alongside Splunk Cloud Platform 10.5.2605 through its documented external APIs or handoffs."13metadata:14 source: https://skillmd.com/skills/chambear2809/splunk-observability-aws-integration15 splunk_cloud_10_5: "not-applicable"16 compatibility_verified: "2026-08-20"17---1819# Splunk Observability Cloud <-> AWS Integration2021## Prerequisites2223| Tool or access | Purpose | Verify |24|---|---|---|25| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |26| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |27| Credential files for live modes | Keep secrets out of chat | Verify paths only |2829## Workflow Overview3031```text32┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐33│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │34└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘35```3637## When to Activate3839- Connect AWS to Splunk Observability Cloud, configure CloudWatch Metric Streams, render IAM policies, manage the40 AWSCloudWatch REST/Terraform object, monitor Bedrock metrics, set up multi-account AWS Organizations onboarding,41 audit drift,.42- Preview and review the splunk observability aws integration workflow before any live apply phase.43- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.4445## Scope4647Follow the documented read-only or render-first path whenever it is available.48This skill does not imply permission to mutate live systems. Require explicit49apply flags, protected credentials, and operator review for state changes.5051## Examples5253Inspect the supported setup modes before selecting one:5455```bash56bash skills/splunk-observability-aws-integration/scripts/setup.sh --help57```5859Expected output: usage, supported modes, and required arguments are displayed60without changing the target environment.6162Inspect validation modes before running completion checks:6364```bash65bash skills/splunk-observability-aws-integration/scripts/validate.sh --help66```6768Expected output: offline, live, and completion options are displayed when the69skill supports them; help exits without mutation.7071## Troubleshooting7273| Issue | Cause | Resolution |74|---|---|---|75| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |76| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |77| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |78| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |7980Render-first skill that owns the entire `AWSCloudWatch` integration object in81Splunk Observability Cloud. The skill is **standalone reusable** and does not82require any other Splunk Observability skill to run, but it hands off to other83skills for adjacent surfaces (logs, Lambda APM, dashboards, detectors, OTel84collectors).8586The workflow is render-first by default. Live API changes only happen when the87user explicitly asks for `--apply`.8889## Coverage Model9091Every rendered section gets an explicit coverage status in92`coverage-report.json`:9394- `api_apply` — a documented public REST API supports create / update / delete /95 validate (e.g. `POST /v2/integration`).96- `api_validate` — a documented public REST API supports read or validation only97 (e.g. preflighting `ec2:DescribeRegions` permission via the live integration's98 reported region list).99- `deeplink` — the skill renders a deterministic Splunk Observability Cloud UI100 link and validates referenced data where an API allows (e.g. the AWS101 guided-setup deeplink for AWS-managed Metric Streams).102- `handoff` — the skill renders deterministic operator steps for cross-skill103 workflows (e.g. logs path -> `splunk-app-install` for `Splunk_TA_AWS`).104- `not_applicable` — the section does not apply to the chosen target (e.g.105 Splunk-managed Metric Streams permissions when `connection.mode` is `polling`,106 or any AWS integration on `realm: us2-gcp` because that realm is GCP-hosted107 and has no AWS STS region mapping).108109## Safety Rules110111- Never ask for Splunk Observability tokens, AWS access keys, or AWS secret112 access keys in conversation.113- Never pass any secret on the command line or as an environment-variable114 prefix.115- Use `--token-file` for the Splunk Observability Cloud admin user API access116 token. Token files must be `chmod 600`. Override only with117 `--allow-loose-token-perms` (emits a WARN — use only for short-lived scratch118 tokens).119- Secret files must always be regular, non-symlink, single-hardlink files no120 larger than 64 KiB containing one non-empty printable-ASCII value with at121 most one trailing LF or CRLF. Readers use `O_NOFOLLOW`,122 bounded double reads, and stable metadata/content fingerprints;123 `--allow-loose-token-perms` relaxes only the mode check.124- Authenticated API calls are pinned to supported HTTPS realm hosts and refuse125 every redirect so `X-SF-Token` cannot be forwarded to another URL.126- For `authentication.mode: security_token` (GovCloud / China), use127 `--aws-access-key-id-file` and `--aws-secret-access-key-file`. Both files128 must be `chmod 600`.129- Reject direct secret flags: `--token`, `--access-token`, `--api-token`,130 `--o11y-token`, `--admin-token`, `--sf-token`, `--external-id`,131 `--aws-access-key-id`, `--aws-secret-access-key`, `--aws-secret-key`,132 `--password`.133- Prefer `SPLUNK_O11Y_REALM` and `SPLUNK_O11Y_TOKEN_FILE` from the repo134 `credentials` file when present; these store only realms and token-file135 paths, never token values.136- Strip every secret from `00-09-*.md`, `apply-plan.json`, `payloads/`,137 `current-state.json`, `state/apply-state.json`, and any other rendered138 artifact on disk. The Splunk Observability admin token is referenced as139 `${SPLUNK_O11Y_TOKEN_FILE}` everywhere, never inlined.140- The renderer FAILs render when the operator passes the deprecated141 `enableLogsSync` field with a clear pointer to the `Splunk_TA_AWS` handoff.142- The renderer FAILs render when `regions: []` (the canonical schema rejects an143 empty list and Splunk highly discourages it because new AWS regions144 auto-onboard and inflate cost).145- `bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_o11y_token`146 helps the user create a token file without exposing the secret in shell147 history.148149## Five-mode UX150151| Mode | Flag | Purpose |152|------|------|---------|153| quickstart | `--quickstart` | Render the common scenario and print the supported SecurityToken apply plus ExternalId handoff. No live changes. |154| render | `--render` (default) | Produces the numbered plan tree under `--output-dir`. Never touches live state. |155| discover | `--discover` | Read-only sweep that polls `GET /v2/integration?type=AWSCloudWatch`, writes `current-state.json`, and emits a `drift-report.md` against the rendered plan. |156| doctor | `--doctor` | Runs the troubleshooting catalog and emits `doctor-report.md` with prioritized fixes and the exact `setup.sh --apply` command for each fix. |157| apply | `--apply [SECTIONS]` | Applies selected sections. SecurityToken integration and regional streams can mutate; IAM, ExternalId two-phase setup, and StackSets fail closed to explicit handoffs. |158159Plus quality-of-life flags:160161- `--quickstart-from-live` — turn the live integration into a `template.observed.yaml`162 the operator can edit; never overwrites their `template.example`.163- `--explain` — print the apply plan in plain English, no API calls; for164 change-management approvals.165- `--rollback <section>` — render (do not auto-run) the reverse-engineered166 commands for steps that have a public reversible API.167- `--list-namespaces` — print the supported AWS service / namespace catalog168 (mirrors `references/namespaces-catalog.md`).169- `--list-recommended-stats` — print the per-namespace per-metric stat catalog170 used by `collect_only_recommended_stats: true`.171- `--privatelink-domain {legacy,new}` — pick `signalfx.com` (default; matches172 current Splunk-published PrivateLink doc) vs `observability.splunkcloud.com`173 PrivateLink hostnames.174- `--cfn-template-url URL` — override the default175 `https://o11y-public.s3.amazonaws.com/aws-cloudformation-templates/release/template_metric_streams_regional.yaml`176 (or the StackSets equivalent when `metric_streams.use_stack_sets: true`).177 Overrides must be absolute HTTPS URLs on port 443, end in `.yaml` or `.yml`,178 and contain no credentials, query, fragment, whitespace, or control179 characters. The rendered AWS CLI command shell-quotes the validated URL.180- `--accept-drift FIELD[,FIELD...]` — needed for `--apply` when discover shows181 the live integration differs from the rendered spec on a field with side182 effects (e.g. flipping `useMetricStreamsSync`).183184## Primary Workflow1851861. Collect non-secret values: realm, integration name, AWS account ID(s),187 IAM role name, regions, services list (or `all_built_in`), connection188 mode (polling vs Splunk-managed Metric Streams vs AWS-managed Metric189 Streams), custom namespaces, multi-account toggle.1901912. Create or update a JSON / YAML spec from `template.example`.1921933. Render and validate:194195 ```bash196 bash skills/splunk-observability-aws-integration/scripts/setup.sh \197 --render \198 --spec skills/splunk-observability-aws-integration/template.example \199 --output-dir splunk-observability-aws-integration-rendered200 ```2012024. Review `splunk-observability-aws-integration-rendered/`:203 - `README.md` — TL;DR and ordered next-step commands.204 - `architecture.mmd` — Mermaid topology of the rendered integration.205 - `00-prerequisites.md` through `09-handoff.md` — numbered per-section plans.206 - `coverage-report.json` — per-section coverage status.207 - `apply-plan.json` — apply ordering with idempotency keys (no secrets).208 - `payloads/` — per-step request bodies for REST calls and CFN parameters.209 - `aws/` — CloudFormation template stubs (regional or StackSets) and210 Terraform `.tf` files for the AWS-side resources and the Splunk-side211 integration object.212 - `iam/` — per-use-case IAM JSON (foundation, polling, streams, tag-sync,213 Cassandra-special-case, GovCloud security-token).214 - `scripts/` — per-step apply scripts and cross-skill handoff drivers.215 - `support-tickets/` — pre-filled tickets when Splunk Support is required.2162175. Apply SecurityToken mode only when explicitly requested. ExternalId mode218 uses the rendered two-phase handoff and direct apply refuses a partial219 disabled integration:220221 The rendered `iam/iam-trust.json` intentionally uses222 `${SPLUNK_AWS_ACCOUNT_ID_FROM_POST_RESPONSE}` until the create response is223 available. After `POST /v2/integration`, replace that placeholder with the224 returned `sfxAwsAccountArn` before deploying the IAM trust policy.225226 ```bash227 bash skills/splunk-observability-aws-integration/scripts/setup.sh \228 --apply integration,validation \229 --spec skills/splunk-observability-aws-integration/template.example \230 --realm us1 \231 --token-file /tmp/splunk_o11y_admin_token \232 --aws-access-key-id-file /tmp/aws_access_key_id \233 --aws-secret-access-key-file /tmp/aws_secret_access_key234 ```235236 To run only a subset of sections:237238 ```bash239 bash skills/splunk-observability-aws-integration/scripts/setup.sh \240 --apply integration \241 --spec my-security-token-aws-integration.yaml \242 --token-file /tmp/splunk_o11y_admin_token \243 --aws-access-key-id-file /tmp/aws_access_key_id \244 --aws-secret-access-key-file /tmp/aws_secret_access_key245 ```246247## Supported Sections248249Specs use `api_version: splunk-observability-aws-integration/v1` and can250include the following top-level blocks (full reference in251[reference.md](reference.md)):252253- `prerequisites` — region/realm preflight, FedRAMP/GovCloud/GCP carve-out,254 IAM permission check stubs, CFN template URL HTTP HEAD probe.255- `authentication` — `external_id` or `security_token` mode, AWS account ID,256 IAM role name, returned external ID.257- `connection` — polling vs Splunk-managed vs AWS-managed vs Terraform-only258 mode; `pollRate`, `metadataPollRate`, `inactiveMetricsPollRate`.259- `regions` — explicit AWS region list (cannot be empty).260- `services` — `all_built_in` / `explicit` / `namespace_filtered` /261 `custom_only`; `collect_only_recommended_stats`; `metric_stats_to_syncs`;262 `namespace_sync_rules` (for built-in `AWS/*` namespaces).263- `custom_namespaces` — `simple_list` (serializes to `customCloudwatchNamespaces`)264 OR `sync_rules` (serializes to `customNamespaceSyncRules`); they conflict.265 `sync_custom_namespaces_only` toggle.266- `guards` — `enable_check_large_volume`, `ignore_all_status_metrics`,267 `sync_load_balancer_target_group_tags`, `enable_aws_usage`.268- `metric_streams` — `use_metric_streams_sync`, `managed_externally`,269 `named_token`, `cloudformation`, `cloudformation_template_url`,270 `use_stack_sets`, `terraform`.271- `private_link` — `enable`, `endpoint_types`, `service_name_overrides`.272- `terraform_provider` — `source` (`splunk-terraform/signalfx`), `version`273 (`~> 9.0` default).274- `multi_account` — `enabled`, `control_account_id`, `member_accounts`,275 `cfn_stacksets`.276- `handoffs` — `lambda_apm`, `logs_via_splunk_ta_aws`, `dashboards`,277 `detectors`, `otel_collector_for_ec2_eks`.278279## Hand-offs to Other Skills280281- AWS log ingestion -> `bash skills/splunk-app-install/scripts/install_app.sh282 --source splunkbase --app-id 1876` (Splunk Add-on for AWS, Splunkbase 1876,283 min v8.1.1). Renderer preflights for `Splunk_TA_amazon_security_lake` and284 emits an uninstall step before the v7+ upgrade.285- Log Observer Connect to surface those logs in O11y ->286 [`splunk-observability-cloud-integration-setup`](../splunk-observability-cloud-integration-setup/SKILL.md).287- Native O11y dashboards for AWS namespaces ->288 [`splunk-observability-dashboard-builder`](../splunk-observability-dashboard-builder/SKILL.md).289- Detectors / AutoDetect alerting ->290 [`splunk-observability-native-ops`](../splunk-observability-native-ops/SKILL.md).291- OTel collector on EC2 / EKS for richer host telemetry than `CWAgent` ->292 [`splunk-observability-otel-collector-setup`](../splunk-observability-otel-collector-setup/SKILL.md).293- Lambda APM via the Splunk OpenTelemetry Lambda layer (publisher294 `254067382080`) ->295 [`splunk-observability-aws-lambda-apm-setup`](../splunk-observability-aws-lambda-apm-setup/SKILL.md).296 Renderer emits a hand-off stub in `09-handoff.md`.297298## Out of Scope299300- AWS log collection via the Splunk AWS log collector Lambda (handed off to301 `Splunk_TA_AWS`; the `enableLogsSync` API field is deprecated and rejected302 by the renderer).303- Lambda APM instrumentation via the OpenTelemetry Lambda layer (handed off to304 [`splunk-observability-aws-lambda-apm-setup`](../splunk-observability-aws-lambda-apm-setup/SKILL.md)).305- Native O11y AWS dashboard / detector CRUD (handed off to306 `splunk-observability-dashboard-builder` and307 `splunk-observability-native-ops`).308- AppDynamics for AWS workloads (separate AppDynamics SaaS workflow).309- Splunk Cloud Platform Data Manager AWS log onboarding (different product;310 Splunk Cloud Platform side, not Splunk Observability Cloud).311312## Validation313314```bash315bash skills/splunk-observability-aws-integration/scripts/validate.sh \316 --output-dir splunk-observability-aws-integration-rendered317```318319Static checks: required-files, IAM JSON shape, and a secrets-leak scan across320every rendered file.321322With `--live`, the CFN HTTPS probe is mandatory only when the rendered plan323uses that template. The live API check requires exactly one enabled integration324matching the rendered name, AWS account, regions, polling/stream mode, services,325custom namespaces, rates, and guards, then calls326`GET /v2/integration/validate/{id}`. Missing credentials, tools, or evidence327fail validation. The validation endpoint requires a short-lived administrator328User API session token.329330For the cross-domain AWS/EKS/O11y production gate, render clean packets with331`scripts/staging/render-aws-eks-o11y.py` and run332`scripts/staging/validate-aws-eks-o11y.sh`. See333[`../../scripts/staging/README.md`](../../scripts/staging/README.md). The gate334is read-only and records a private, sanitized JSON report.335336See `reference.md` and the focused references under `references/` for full337detail. The plan and corrections that produced this skill are recorded in338`.cursor/plans/splunk_o11y_aws_integration_skill_*.plan.md`.