Dd AWS Integration

Datadog AWS Integration

You are helping a user set up the Datadog AWS integration using Terraform.

The integration creates an IAM role in the customer's AWS account that Datadog assumes via cross-account role delegation. Datadog's AWS account is granted sts:AssumeRole with an external ID for security. No long-lived credentials are stored - Datadog assumes the role on demand.

This is a hands-on setup: run the commands yourself as part of the conversation rather than handing the user a list, keep them in the loop, and pause for confirmation before terraform apply.

Phase 0: Preflight

Terraform or OpenTofu. Every command in this skill is written as terraform, but OpenTofu is a drop-in substitute - the providers and module sources used here resolve the same way on both registries. Check which binary the user actually has before Phase 1:

command -v terraform tofu

If only tofu is on the PATH, read every terraform <subcommand> below as tofu <subcommand>. If both are present, ask which one the user wants rather than guessing.

Datadog credentials. Load DD_SITE / DD_API_KEY / DD_APP_KEY from the environment (falling back to .env.local / .env) and validate both keys - they fail independently:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["''']//;s/["''']$//'); [ -n "$v" ] && export "$k=$v"; done; done
: "${DD_SITE:=datadoghq.com}"
echo "DD_SITE=${DD_SITE}"
echo "DD_API_KEY=$([ -n "${DD_API_KEY:-}" ] && echo set || echo UNSET)   DD_APP_KEY=$([ -n "${DD_APP_KEY:-}" ] && echo set || echo UNSET)"
printf 'DD-API-KEY: %s\n' "$DD_API_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "validate:     HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v1/validate"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "current_user: HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v2/current_user"

App URL. The Datadog app host is not app.${DD_SITE} for every site. It is https://app.datadoghq.com (US1), https://app.datadoghq.eu (EU1), https://app.ddog-gov.com (Gov), and for every other site it is https://${DD_SITE} itself - https://us3.datadoghq.com, https://us5.datadoghq.com, https://ap1.datadoghq.com, https://ap2.datadoghq.com, https://uk1.datadoghq.com. Resolve it once and substitute it wherever <APP_BASE> appears below. Full list: https://docs.datadoghq.com/getting_started/site/

Remember the resolved DD_SITE. Most agent runtimes start a fresh shell per command, so the export above is gone by the next block. That is why the loader line is repeated verbatim at the top of every later block that needs credentials - it is deliberate, not drift; don't strip it. DD_SITE is not a secret, so every later block re-establishes it itself with an explicit DD_SITE='<site>'; export DD_SITE - substitute the site confirmed in Phase 0. It is a plain assignment rather than : "${DD_SITE:=...}" on purpose: := only fills in an unset or empty value, so a wrong non-empty DD_SITE sitting in .env would survive it and every call would go to the wrong region. The keys are guarded with :? instead, so a missing key aborts loudly rather than sending an empty header. Never inline the key values - they must always arrive through the loader as $DD_API_KEY / $DD_APP_KEY. Two consequences follow, and both are deliberate:

• Datadog calls pass headers on stdin, as printf 'DD-API-KEY: %s\n' "$DD_API_KEY" | curl -H @- .... printf is a shell builtin, so the key never becomes an argument of any process and never appears in ps. Writing -H "DD-API-KEY: $DD_API_KEY" instead would put it in curl's argv. (-H @- needs curl 7.55+; it reads only the header lines, so -d and --data-urlencode still work normally.)
• Terraform never receives the keys as values at all for AWS, Azure, and GCP: the Datadog provider reads DD_API_KEY / DD_APP_KEY from the environment, so there are no root variables, no -var= arguments, and nothing for Terraform to record in state or a saved plan. (OCI is the exception - its module needs them as inputs, so there they travel as TF_VAR_*.)

Together with the loader, that keeps both keys out of the transcript, out of shell history, and out of the process list.

Tools. terraform is required; the aws CLI is optional (it only looks up the account ID):

command -v terraform || echo "MISSING terraform - https://developer.hashicorp.com/terraform/install"
command -v aws >/dev/null 2>&1 && echo "aws: available" || echo "aws: not installed"

The snippets here are POSIX shell. Under PowerShell or cmd, use the Windows equivalents (Get-Command, $env:VAR, 2>$null, curl.exe) - same calls, same order.

Phase 1: Determine Scope

Ask the user for:

• Their AWS account ID (12-digit number).
• Which AWS regions they want to monitor. Default to all regions if they have no preference.

If they don't know their account ID:

If the aws CLI is available:

aws sts get-caller-identity --query "Account" --output text

Otherwise, offer to install it - it's the fastest way to look this up automatically:

I can read your AWS account ID directly if you install the AWS CLI. On macOS: brew install awscli. Otherwise: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html. After install, run aws configure (or set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars). Want to install it now, or paste your account ID yourself?

If the user prefers to find it manually: AWS Console top-right account menu, or My Account at https://console.aws.amazon.com/billing/home#/account.

Also determine the correct Datadog trusted AWS account ID. It depends on two things - the Datadog site and the AWS partition the monitored account lives in - and getting it wrong writes an sts:AssumeRole trust for the wrong account, so do not guess.

For a commercial AWS account (partition aws):

us2.ddog-gov.com (US2-FED) is deliberately not in that table, and neither is any GovCloud or China AWS account. Datadog publishes a different id for a GovCloud-partition account than for a commercial account on the same government site, and this skill does not carry those values. In any of those cases, stop and read the id from Datadog's AWS manual setup guide with the DATADOG SITE selector on that page set to the user's site - the page renders the id per site, and distinguishes the commercial value from the GovCloud one. Never fall back to the commercial default for a site or partition that isn't listed above.

Phase 2: Generate and Apply Terraform

Check what already exists - local Terraform first, then Datadog

Do this before generating or applying anything, and in this order. Local state first, because it decides whether an existing integration is something you can update or something you must not touch:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print
# A project is "present" if it has configuration - .terraform/ may simply not exist yet on a fresh clone
# with a remote backend, and terraform.tfstate does not exist at all when state is remote.
if [ -n "$(find . -maxdepth 1 -type f \( -name '*.tf' -o -name '*.tf.json' \) -print -quit)" ]; then
  terraform init -input=false >/dev/null || { echo "terraform init failed - resolve that before concluding anything about existing state"; exit 1; }
  out=$(terraform state list 2>&1); rc=$?
  if [ "$rc" -ne 0 ]; then
    case $out in
      *'No state file'*|*'no state'*|*'Backend initialization required'*)
        echo "project is initialized but has no state yet - treat as a clean install" ;;
      *)
        printf '%s\n' "$out"
        echo "could not read state (backend or credentials problem) - do NOT treat this as 'nothing exists'"; exit 1 ;;
    esac
  elif [ -z "$out" ]; then
    echo "state is empty - treat as a clean install"
  else
    printf '%s\n' "$out" | grep -F 'datadog_integration_aws_account' || echo "state exists but holds no datadog_integration_aws_account resource"
  fi
else
  echo "no Terraform configuration here yet - clean install"
fi

Match the exact resource address datadog_integration_aws_account, not a loose grep -i datadog: unrelated Datadog resources, or cloud IAM left behind by a partial apply, would otherwise read as a managed integration.

Then ask Datadog what it already has:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["''']//;s/["''']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -X GET -H @- "https://api.${DD_SITE}/api/v2/integration/aws/accounts")
code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
[ "$code" = "200" ] || { echo "lookup failed with HTTP $code - do not assume 'not connected':"; printf '%s\n' "$body"; exit 1; }
printf '%s\n' "$body"

Now reconcile the two answers before doing anything:

• Not in Datadog, nothing in local state - a clean install. Continue.

• In Datadog and present in local state - this is the update/repair case, not a duplicate. Continue into the Terraform below as a change to the existing resources, and let the plan show what it will alter.

• In Datadog but absent from local state (the AWS account ID from Phase 1 already appears) - stop. Applying would either create a duplicate or fight with whatever manages it. Say so plainly and offer the options: import the existing object into this project (terraform import datadog_integration_aws_account.datadog_integration "<config-id>", where the config id is not the AWS account id - get it from the List all AWS integrations endpoint, querying by AWS account id), manage it where it is already managed, or delete it in Datadog first. Only continue if the user picks one and confirms. Two things about importing, in this order. Generate the configuration first (the Terraform below, adapted to the identity that already exists - same role/app/service-account name), because terraform import binds an existing object to a configured resource address and fails without one. And importing the Datadog registration alone is not enough: the cloud-side identity (the IAM role, the app registration, the service account) is still outside state, so either import those too or reference them with data sources, or the next apply will try to create them again and collide.

  # 1. get the Datadog-side config id (NOT the AWS account id)
  for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["''']//;s/["''']$//'); [ -n "$v" ] && export "$k=$v"; done; done
  DD_SITE='datadoghq.com'; export DD_SITE   # <- the site confirmed in Phase 0
  : "${DD_API_KEY:?}"; : "${DD_APP_KEY:?}"
  printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
    | curl -sS -H @- "https://api.${DD_SITE}/api/v2/integration/aws/accounts"
  # 2. import it, after the configuration below exists
  terraform import datadog_integration_aws_account.datadog_integration "<config-id-from-above>"
  

• Present in local state but absent from Datadog - a partial or rolled-back install. The cloud-side resources may exist while the registration does not. Do not start from scratch: run the plan and let it show what is missing, and expect it to re-create only the registration.

Check for Existing Terraform

Before generating a new Terraform configuration, check if the user already has a Terraform project in the current directory or nearby:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print

If existing .tf files are found:

• Read them to understand what providers and resources are already configured.
• If a datadog provider already exists, reuse its configuration - do not create a duplicate.
• If an aws provider already exists, reuse it.
• Only add the new resources needed (IAM role, policies, datadog_integration_aws_account) to the existing project. Do not regenerate providers, variables, or terraform blocks that already exist.
• If the user has a modular layout (e.g., separate files per concern), create a new file like datadog-aws-integration.tf for the Datadog resources.

If no existing Terraform is found, generate a standalone configuration.

The full HCL template - providers, the trust policy, the dynamically-fetched permission set with its 6144-character chunking, the role, and the datadog_integration_aws_account registration - is in references/terraform.md. Read it now and emit it with the placeholders filled in.

The template leaves logs_config.lambda_forwarder empty (metrics only). Forwarding AWS logs to Datadog is a separate follow-on that deploys the Datadog Forwarder Lambda and registers its ARN here; point the user at https://docs.datadoghq.com/logs/guide/forwarder/ once the integration is live. Don't set up the forwarder inline.

Applying the Terraform

1. Replace all <PLACEHOLDER> values:

   • <DD_SITE> from Phase 0
   • <DATADOG_TRUSTED_ACCOUNT_ID> from the table in Phase 1
   • <AWS_ACCOUNT_ID> from Phase 1

2. Ensure the user has AWS credentials configured (aws configure or environment variables), and that they belong to the account being registered. If they don't, Terraform creates the IAM role in one account while registering a different one with Datadog - the apply succeeds and the integration is broken:

   want='<AWS_ACCOUNT_ID>'   # from Phase 1
   have=$(aws sts get-caller-identity --query Account --output text) || { echo "no usable AWS credentials"; exit 1; }
   [ "$have" = "$want" ] || { echo "ambient AWS credentials are for account $have, but you are registering $want - stop and fix this"; exit 1; }
   echo "AWS credentials match account $want"
   

   Without the aws CLI, the template enforces the same thing with a lifecycle.precondition on the IAM role, which fails the apply. (A check block would only warn - Terraform does not stop an apply for a failed check.)

3. Run terraform init to install providers.

4. Plan, and save the plan to a file. The Datadog provider reads DD_API_KEY / DD_APP_KEY straight from the environment, so there are no root variables and no -var= arguments - nothing secret ends up in the plan file, in state, or on a command line:

   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["''']//;s/["''']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   umask 077          # tighten permissions on the plan file anyway
   terraform plan -out=tfplan
   

   Show the plan output to the user and wait for explicit confirmation.

5. Apply that saved plan, only after the user confirms it. Applying the file is what makes the approval meaningful: terraform apply with no plan file computes a brand-new plan, and -auto-approve would execute it without anyone seeing it, so anything changed since the plan would go in unreviewed:

   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["''']//;s/["''']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   trap 'rm -f tfplan' EXIT HUP INT TERM   # the plan file goes away even if this is interrupted
   if terraform apply tfplan; then
     echo "apply complete"
   else
     echo "terraform apply FAILED - do NOT verify or report success"
     exit 1
   fi
   

   The trap removes the plan file on every exit path, including Ctrl-C while the user is deciding. The if/else around the apply matters because a cleanup command as the block's last line would make a failed apply exit 0, and the agent would go on to "verify" a deployment that never happened. (It is an if rather than status=$? on purpose: status is a read-only variable in zsh.)

   The plan file holds no key material at all, because the keys never become Terraform values - the provider reads them from the environment. That is what makes -out safe here, on any Terraform version, and it is why an existing project needs no variable changes either.

6. After terraform apply succeeds, verify the integration registered with Datadog:

   for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["''']//;s/["''']$//'); [ -n "$v" ] && export "$k=$v"; done; done
   DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
   # Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
   : "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
   resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
     | curl -sS -w '\n%{http_code}' -X GET -H @- "https://api.${DD_SITE}/api/v2/integration/aws/accounts")
   code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
   [ "$code" = "200" ] || { echo "lookup failed with HTTP $code - do not assume 'not connected':"; printf '%s\n' "$body"; exit 1; }
   printf '%s\n' "$body"
   

   Confirm the response includes the AWS account ID that was just provisioned. If the account is missing, surface the response to the user so they can debug.

Getting the Most Out of Your Integration

Once terraform apply completes successfully, congratulate the user and let them know metrics typically arrive within 5-10 minutes. Then check for early metrics and show a widget.

Checking for Metrics

Give it a few seconds, then query the metrics API - substitute the account ID from Phase 1 for <AWS_ACCOUNT_ID>:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["''']//;s/["''']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
sleep 10
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -G -H @- "https://api.${DD_SITE}/api/v1/query" \
  --data-urlencode "from=$(($(date +%s) - 900))" \
  --data-urlencode "to=$(date +%s)" \
  --data-urlencode "query=avg:aws.ec2.cpuutilization{aws_account:<AWS_ACCOUNT_ID>} by {host}")
code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
[ "$code" = "200" ] || { echo "metric query failed with HTTP $code - that is NOT 'metrics still propagating':"; printf '%s\n' "$body"; exit 1; }
printf '%s\n' "$body"

The response carries the series/pointlist JSON the widget rendering below expects:

Rendering the Widget

If the series array is non-empty, render an ASCII chart from the real data:

• Use the pointlist values to plot the line, scaling Y-axis to actual min/max.
• Use box-drawing characters (╭, ╰, ─, │, ┤) for the line.
• List the host names from each series scope at the bottom.
• Show the top 3 series by average value if multiple are returned.

If the series array is empty, show this static preview instead and let the user know metrics are still propagating:

┌─────────────────────────────────────────────────────────┐
│  aws.ec2.cpuutilization          ▂▃▅▆▇▆▅▃▂▁▂▃▅▆▇█▇▅▃  │
│  100% ┤                                          ╭──╮   │
│   75% ┤                    ╭───╮              ╭──╯  │   │
│   50% ┤              ╭────╯   ╰──╮     ╭────╯     │   │
│   25% ┤    ╭────────╯            ╰────╯           │   │
│    0% ┤────╯                                       │   │
│       └────────────────────────────────────────────┘   │
│                                                         │
│  Metrics are on their way - check back in a few minutes │
└─────────────────────────────────────────────────────────┘
Metrics Explorer: <APP_BASE>/metric/explorer?exp_metric=aws.ec2.cpuutilization

Confirm to the user that their integration is configured and data will appear shortly. All links use DD_SITE - construct them as <APP_BASE>/....

• AWS Integration tile: <APP_BASE>/integrations/amazon-web-services - access the pre-built dashboard and verify the integration is active.
• Metrics Explorer: <APP_BASE>/metric/explorer?exp_metric=aws.ec2.cpuutilization - confirm data is flowing.
• Each AWS service (EC2, RDS, Lambda, S3, ECS, EKS, etc.) has its own dashboard that activates automatically when metrics for that service are detected.

Recommended Monitors - suggest creating monitors at <APP_BASE>/monitors/create:

• EC2 CPU utilization exceeding a threshold

• RDS free storage space running low

• Lambda error rate spikes

• ELB unhealthy host count

• Cloud Security: <APP_BASE>/security/compliance - review security posture findings across AWS resources.

• Resource Catalog: <APP_BASE>/infrastructure/catalog - browse EC2 instances, RDS databases, Lambda functions, and more.

• Infrastructure Map: <APP_BASE>/infrastructure/map - visualize AWS infrastructure.

Explore more Datadog products:

• Log Management: <APP_BASE>/logs - centralized log search and alerting.
• APM & Traces: <APP_BASE>/apm/getting-started - distributed tracing for applications on Lambda, ECS, EKS, or EC2.
• Notebooks: <APP_BASE>/notebook - shareable investigations combining metrics, logs, and events.

Important Notes

• The user must have IAM permissions to create roles, policies, and policy attachments in their AWS account.
• The external ID is generated by Datadog and included automatically in the terraform - it should not be hardcoded.
• The IAM permissions are fetched dynamically from Datadog via datadog_integration_aws_iam_permissions - they may change over time as Datadog adds new integrations.
• Permissions are automatically split into multiple policies to stay under the AWS 6144-character IAM policy size limit.
• The Datadog API and app keys are never passed to Terraform as values: the datadog provider reads DD_API_KEY and DD_APP_KEY from the environment, so there are no root variables, no -var= arguments, and nothing for Terraform to record in state or a saved plan. Don't declare key variables, and don't write the keys into a committed .tfvars file or any other persistent file.
• Never run terraform apply without showing the plan to the user first.