Splunk Observability AWS Lambda Apm Setup

Splunk Observability Cloud — AWS Lambda APM Setup

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Instrument Lambda functions for APM/tracing, attach the Splunk OTel Lambda layer, wire SPLUNK_ACCESS_TOKEN safely, set up Lambda APM dashboards or detectors, or migrate from Datadog, New Relic, or ADOT to Splunk OTel.
• Preview and review the splunk observability aws lambda apm setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Render-first skill that owns the complete lifecycle of Splunk OTel Lambda layer attachment. The workflow is render-first by default. AWS Lambda function changes only happen when the operator explicitly runs --apply after reviewing the rendered plan.

The skill is the live fulfillment of the handoffs.lambda_apm stub emitted by splunk-observability-aws-integration (previously marked "future companion").

Coverage Model

Safety Rules

• Never ask for the Splunk O11y access token in conversation.
• Never pass SPLUNK_ACCESS_TOKEN as a command-line value. Use --token-file. The apply helper rotates the selected AWS secret, fetches it into a private temporary file, and merges it into the Lambda environment without exposing it in rendered files, shell history, or process arguments.
• Token files must be regular, single-hardlink files containing one non-empty printable-ASCII value with at most one trailing LF or CRLF, and no larger than the backend limit (4 KiB for default-tier SSM Parameter Store; 64 KiB for Secrets Manager) with mode 600. The generated writer opens the file once with O_NOFOLLOW, verifies stable inode, metadata, and content fingerprints, and builds the AWS payload from that descriptor. Use write_secret_file.sh to create one without shell-history exposure.
• --allow-loose-token-perms is an explicit development-only escape that relaxes only the source mode check. The setup wrapper propagates it to the generated writer; regular-file, non-symlink, single-hardlink, size, UTF-8, and fingerprint checks remain mandatory.
• Reject direct-secret flags: --token, --access-token, --api-token, --o11y-token, --sf-token, --password.
• signalfx/splunk-otel-lambda is BETA. Gate all operations behind accept_beta: true in the spec or --accept-beta on the CLI.
• GovCloud (us-gov-*) and China (cn-*) regions have no published layers. The renderer fails with not_applicable coverage; do not fabricate ARNs.

Five-mode UX

--quickstart-from-live --target FUNCTION writes only an allowlisted subset of the Lambda configuration to a mode-600 snapshot. It deliberately excludes Environment (and therefore environment secrets), revision IDs, and other unneeded response fields.

Primary Workflow

1. Copy and edit the spec

cp skills/splunk-observability-aws-lambda-apm-setup/template.example my-lambda-spec.yaml
# fill in realm, targets (function names, regions, runtimes, arches)

2. Render

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --render \
  --spec my-lambda-spec.yaml \
  --realm us1

3. Review the plan

splunk-observability-aws-lambda-apm-rendered/
  01-overview.md               # plan summary + apply command
  02-targets.md                # per-function targets + resolved layer ARNs
  03-layers.md                 # layer attachment plan
  04-env.md                    # environment variable plan
  05-validation.md             # validation steps
  aws-cli/apply-plan.sh        # guarded AWS CLI apply helper
  terraform/main.tf            # Terraform resource snippets
  cloudformation/snippets.yaml # CloudFormation snippets
  sam/template.yaml            # AWS SAM template snippet
  cdk/lambda-apm-stack.ts      # CDK TypeScript snippet
  cdk/lambda_apm_stack.py      # CDK Python snippet
  sar/README.md                # SAR advisory (Splunk does not publish SAR)
  container-image/             # Dockerfile.<runtime> (image-package targets only)
  iam/                         # IAM policy (local collector disabled only)
  scripts/write-splunk-token.sh # one-time token write to secret backend
  scripts/handoffs.sh           # cross-skill handoff drivers
  coverage-report.json          # per-section coverage status

4. Write the access token to the secret backend (once)

# Prepare the token file (never put the value in shell history):
bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_o11y_token

# Write token to Secrets Manager (or SSM if secret_backend: ssm):
TOKEN_FILE=/tmp/splunk_o11y_token \
  bash splunk-observability-aws-lambda-apm-rendered/scripts/write-splunk-token.sh

# Keep the local file until the apply step below completes, then delete it.

5. Apply

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --apply \
  --spec my-lambda-spec.yaml \
  --realm us1 \
  --token-file /tmp/splunk_o11y_token

rm /tmp/splunk_o11y_token

Or apply only specific sections:

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --apply layer,env \
  --spec my-lambda-spec.yaml

Quickstart (single command)

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --quickstart \
  --spec my-lambda-spec.yaml \
  --realm us1 \
  --accept-beta

The quickstart renders the plan and prints the exact --apply command to run next. No AWS calls happen until you explicitly run --apply.

GitOps mode (Terraform / CloudFormation only)

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --render \
  --gitops-mode \
  --spec my-lambda-spec.yaml

--gitops-mode suppresses the aws-cli/ directory and produces only terraform/ and cloudformation/ artifacts.

The Terraform variant necessarily reads the token as plaintext and stores it in Terraform state as part of the Lambda environment. Use only an encrypted, access-controlled remote backend with locking and never commit local state; prefer the CloudFormation dynamic-reference or guarded AWS CLI variant when state cannot be protected to the same standard as the token.

Doctor

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --doctor \
  --target my-function \
  --realm us1

Doctor checks: beta status, snapshot freshness, vendor conflicts (Datadog / New Relic / AppDynamics / Dynatrace), ADOT conflict, X-Ray coexistence flag, and VPC egress reminder when local_collector_enabled: false.

Rollback

# Detach layer only:
bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --rollback layer \
  --target my-function

# Remove env vars only:
bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --rollback env \
  --target my-function

# Detach IAM ingest-egress policy (only when local_collector_enabled=false):
bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --rollback iam \
  --target my-function

# Full rollback (layer → env → iam in sequence):
bash skills/splunk-observability-aws-lambda-apm-setup/scripts/setup.sh \
  --rollback all \
  --target my-function

Each rollback section renders idempotent AWS CLI commands for review; the skill never mutates AWS without operator review.

Hand-offs

• CloudWatch Lambda metrics → splunk-observability-aws-integration
• Lambda APM dashboards → splunk-observability-dashboard-builder
• Cold-start / error / latency detectors → splunk-observability-native-ops
• Lambda log ingestion → splunk-connect-for-otlp-setup
• Gateway OTel Collector path → splunk-observability-otel-collector-setup (set handoffs.gateway_otel_collector: true and local_collector_enabled: false)
• FROM splunk-observability-aws-integration → this skill is the fulfillment of handoffs.lambda_apm: true

Out of Scope

• Lambda log ingestion (handed off to splunk-connect-for-otlp-setup)
• CloudWatch metrics for AWS/Lambda namespace (handed off to splunk-observability-aws-integration)
• AppDynamics serverless instrumentation (handled by splunk-appdynamics-apm-setup)
• AlwaysOn Profiling for Lambda (not officially supported in signalfx/splunk-otel-lambda)
• Go, Ruby, .NET Lambda runtimes (no published layers from publisher 254067382080)
• GovCloud / China regions (no published layers; renderer refuses with not_applicable)
• Lambda@Edge (no env-var injection support in the Lambda@Edge execution model; renderer refuses)

Validation

bash skills/splunk-observability-aws-lambda-apm-setup/scripts/validate.sh \
  --output-dir splunk-observability-aws-lambda-apm-rendered

Static checks: required files, IAM JSON shape, secret-leak scan. With --live: runs an unauthenticated reachability-only probe of the Splunk ingest endpoint. It does not validate Lambda configuration, token authorization, span export, or APM data arrival. Invoke the function and verify its expected span attributes in APM for configured-state acceptance.

See references/splunk-doc-feature-matrix.md for the full feature matrix and limitations.