AWS Cost Investigation

AWS Cost Investigation

Operational skill for diagnosing AWS cost spikes and auditing for ongoing waste. The focus is on diagnostic flow (data-first, not guess-first) plus a concrete trap inventory with detection CLI for each.

When to invoke

Symptoms:

• "The bill is up $X with no deploys" / "AWS bill spiked last month."
• An anomaly notification from AWS Cost Anomaly Detection.
• Cost Explorer dashboards show large (no tag) slices despite tagging policies.
• NAT Gateway charges growing month over month.
• Looking at a Savings Plan / Reserved Instance commitment decision.
• A general account audit ("find the waste").
• Designing a cost-allocation tagging strategy.

Cross-cutting rules

1. Data first, guesses never. When asked to diagnose a spike, the first action is to query Cost Explorer. Do NOT guess "probably S3" or "probably NAT" without numbers. Naming a likely culprit without data is anti-pattern #1.
2. Compare windows of equal length. A 7-day spike compares to the prior 7 days, not month-to-date. A monthly spike compares to the same days of the prior month, not the full prior month.
3. Never quote a specific dollar amount as a pricing fact. AWS prices change. State relative magnitudes (Gateway endpoints are free; Interface endpoints are cheaper than NAT for high egress) and link to the AWS pricing page for current numbers when a precise answer is needed.
4. Activation is the silent gate for tag-based analysis. A tag on a resource is invisible to Cost Explorer / CUR until it's activated as a Cost Allocation Tag. See Cost allocation tagging.
5. Stop-the-bleeding before re-architecting. When a runaway cost is identified, the first action is to cap it (set retention, delete idle, add a budget alarm) — not redesign the workload.

Diagnostic flow for spikes

Run the steps in order. Don't skip ahead.

Step 1 — Service-level diff

# Compare two equal-length windows (here: last 7d vs prior 7d)
END=$(date -u +%Y-%m-%d)
START=$(date -u -d '-7 days' +%Y-%m-%d)
PRIOR_END=$START
PRIOR_START=$(date -u -d '-14 days' +%Y-%m-%d)

for window in "$PRIOR_START $PRIOR_END" "$START $END"; do
  read s e <<<"$window"
  aws ce get-cost-and-usage \
    --time-period Start=$s,End=$e \
    --granularity DAILY \
    --metrics UnblendedCost \
    --group-by Type=DIMENSION,Key=SERVICE \
    --query 'ResultsByTime[].Groups[]' \
    --output json > /tmp/ce-$s.json
done
# Diff per-service totals between the two files to find the top growers.

Identify the top-growing service. Do NOT proceed to step 2 without this.

Step 2 — Drill into the top service by usage type

aws ce get-cost-and-usage \
  --time-period Start=$START,End=$END \
  --granularity DAILY \
  --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Elastic Compute Cloud - Compute"]}}' \
  --group-by Type=DIMENSION,Key=USAGE_TYPE

Usage-type taxonomy worth recognizing:

Step 3 — Resource-level attribution (opt-in)

aws ce get-cost-and-usage-with-resources returns per-resource costs but requires:

• Opt-in via Cost Explorer Settings page (one-time).
• Daily granularity for everything except EC2 (which supports hourly).
• 14-day lookback window only.

aws ce get-cost-and-usage-with-resources \
  --time-period Start=$START,End=$END \
  --granularity DAILY \
  --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Elastic Compute Cloud - Compute"]}}' \
  --group-by Type=DIMENSION,Key=RESOURCE_ID

For costs older than 14 days, the CUR is the right tool — see CUR + Athena.

Step 4 — Tag axis (if cost-allocation tags are activated)

aws ce get-cost-and-usage \
  --time-period Start=$START,End=$END \
  --granularity DAILY \
  --metrics UnblendedCost \
  --group-by Type=TAG,Key=team Type=DIMENSION,Key=SERVICE

If the response shows large (no tag) slices, the tag isn't activated — see Cost allocation tagging.

Step 5 — Sanity-check against AWS's anomaly detector

aws ce get-anomalies \
  --date-interval StartDate=$PRIOR_START,EndDate=$END

AWS Cost Anomaly Detection runs its own ML and may have already flagged the cause with RootCauses populated. Free service; underused.

The cost-trap inventory

These are recurring sources of silent cost growth. Audit each in any account you don't know well.

CloudWatch log groups with no retention

Lambda, ECS, CodeBuild, API Gateway all auto-create log groups with retention null = never expire.

aws logs describe-log-groups \
  --query 'logGroups[?retentionInDays==`null`].[logGroupName,storedBytes]' \
  --output table

# Fix: set retention org-wide (example: 30 days)
aws logs describe-log-groups --query 'logGroups[].logGroupName' --output text \
  | xargs -n1 -I{} aws logs put-retention-policy --log-group-name {} --retention-in-days 30

Going forward, Terraform pattern:

resource "aws_cloudwatch_log_group" "lambda" {
  name              = "/aws/lambda/${aws_lambda_function.x.function_name}"
  retention_in_days = 30
  # Create this resource BEFORE the Lambda — otherwise Lambda creates the group with null retention.
}

Unattached EBS volumes

aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' \
  --output table

Volumes in available state bill at full storage cost. Common after EC2 termination with "delete on termination" not set. Snapshot then delete, or just delete if not needed.

Old EBS snapshots (no lifecycle)

aws ec2 describe-snapshots --owner-ids self \
  --query 'Snapshots[?StartTime<=`2024-01-01`].[SnapshotId,StartTime,VolumeSize,Description]' \
  --output table

Snapshots accumulate. Use Data Lifecycle Manager (DLM) to set automated retention going forward; clean up old ones one-time.

Unattached Elastic IPs

aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId,Domain]' \
  --output table

EIPs not attached to a running instance bill hourly. After AWS's 2024 pricing change, attached EIPs also bill — but unattached is always wasted.

Idle Load Balancers

# ALBs and NLBs with low request count over the past 7 days
aws elbv2 describe-load-balancers \
  --query 'LoadBalancers[].[LoadBalancerArn,LoadBalancerName,Type]' \
  --output text \
  | while read arn name type; do
    requests=$(aws cloudwatch get-metric-statistics \
      --namespace AWS/ApplicationELB \
      --metric-name RequestCount \
      --dimensions Name=LoadBalancer,Value=$(echo $arn | cut -d/ -f2-) \
      --start-time $(date -u -d '-7 days' +%Y-%m-%dT%H:%M:%SZ) \
      --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
      --period 604800 --statistics Sum \
      --query 'Datapoints[0].Sum' --output text)
    echo "$name $type $requests"
  done

LBs with near-zero traffic are usually leftover from migrated/deleted services.

NAT Gateway data egress

The single biggest silent killer for workloads in private subnets. Diagnostic:

# How many bytes did NAT GW process this month?
aws cloudwatch get-metric-statistics \
  --namespace AWS/NATGateway \
  --metric-name BytesOutToDestination \
  --start-time $(date -u -d '-30 days' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 86400 --statistics Sum \
  --dimensions Name=NatGatewayId,Value=<natgw-id>

Fix: route the traffic through VPC endpoints. See NAT vs VPC endpoint.

S3 incomplete multipart uploads

Failed/abandoned uploads bill at full storage cost indefinitely. No CLI lists "incomplete uploads across all buckets" — must check per-bucket:

aws s3api list-multipart-uploads --bucket <bucket> \
  --query 'Uploads[?Initiated<=`2024-01-01`].[Key,UploadId,Initiated]'

Fix going forward: lifecycle rule per bucket:

{
  "Rules": [{
    "ID": "abort-incomplete-mpu",
    "Status": "Enabled",
    "Filter": {},
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
  }]
}

gp2 → gp3 migration

gp3 is cheaper than gp2 at the same baseline performance (3000 IOPS / 125 MB/s included). Accounts pre-2021 may still default to gp2. Detection:

aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[].[VolumeId,Size,State,AvailabilityZone]' \
  --output table

# In-place migration (no downtime, online)
aws ec2 modify-volume --volume-id <vol-id> --volume-type gp3

Idle RDS / Aurora

# Instances with zero active connections for a week
aws rds describe-db-instances --query 'DBInstances[].[DBInstanceIdentifier,DBInstanceClass,Engine]' --output text \
  | while read id class engine; do
    conns=$(aws cloudwatch get-metric-statistics \
      --namespace AWS/RDS \
      --metric-name DatabaseConnections \
      --dimensions Name=DBInstanceIdentifier,Value=$id \
      --start-time $(date -u -d '-7 days' +%Y-%m-%dT%H:%M:%SZ) \
      --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
      --period 604800 --statistics Maximum \
      --query 'Datapoints[0].Maximum' --output text)
    echo "$id $class $engine $conns"
  done

Look for 0 or 0.0. Common in dev/staging accounts.

Provisioned-mode DynamoDB with low utilization

DDB Provisioned with auto-scaling can still over-provision. If WCU/RCU utilization is consistently <50%, switch to On-Demand. Diagnostic via CloudWatch ConsumedWriteCapacityUnits / ProvisionedWriteCapacityUnits ratio.

Cross-region data transfer (accidental)

Symptom: a DataTransfer-Regional-Bytes or DataTransfer-OutBytes line item in Cost Explorer with no obvious source. Common causes:

• S3 replication misconfigured (bucket replicating to itself or to a region it shouldn't).
• CloudWatch cross-account logs.
• VPC peering between regions used heavily by an unintended service.

Drill: filter Cost Explorer by USAGE_TYPE matching *DataTransfer* and group by REGION + LINKED_ACCOUNT.

NAT vs VPC endpoint

The decision rule that consolidates many cost issues into one trade-off.

Decision rule:

• S3 or DynamoDB egress from a VPC: always use the Gateway endpoint. Free, no downside.
• ECR pulls, Secrets Manager, SSM, CloudWatch Logs: use Interface endpoints if egress volume is non-trivial. Crossover is usually tens of GB/month — below that, NAT is cheaper because Interface endpoints have a fixed hourly charge.
• Random outbound HTTPS to third-party APIs: stays on NAT. No endpoint exists.

ECR specifically requires three endpoints for image pulls to fully bypass NAT:

1. com.amazonaws.<region>.ecr.api — Interface, for authentication
2. com.amazonaws.<region>.ecr.dkr — Interface, for image-pull operations
3. com.amazonaws.<region>.s3 — Gateway, because ECR stores layers in S3 and the pull hits S3 directly

Without all three, the pull falls back to NAT for the missing leg.

Gateway endpoint setup:

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-aaa rtb-bbb
# Associate with all route tables for subnets that should use the endpoint.

Interface endpoint setup:

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxx \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.us-east-1.ecr.api \
  --subnet-ids subnet-aaa subnet-bbb \
  --security-group-ids sg-xxx \
  --private-dns-enabled
# Private DNS Names must be enabled so the standard AWS DNS names resolve to the endpoint.

Security group on the endpoint must allow 443 from the source security groups.

Commitment decision: Savings Plans vs Reserved Instances

Modern AWS commitment landscape (in order of flexibility, lowest discount first):

Decision rule:

• Baseline EC2/Fargate/Lambda compute that runs 24/7: Compute Savings Plan, 1-year, No Upfront. Highest flexibility, ~25–30% discount range (verify current).
• RDS, ElastiCache, Redshift, OpenSearch: Standard RIs. Savings Plans don't cover these.
• Predictable instance family but flexible region/size: EC2 Instance SP.
• Don't commit at >70% of steady-state. A SP/RI you don't use is pure waste — you pay the committed rate whether you use the capacity or not. Run aws ce get-savings-plans-utilization on existing commitments before buying more.

Term length: 1-year is the default for new commitments. 3-year only when the workload is provably stable and won't migrate (no impending platform change, no architecture rewrites scheduled).

# Check existing SP utilization
aws ce get-savings-plans-utilization \
  --time-period Start=$(date -u -d '-30 days' +%Y-%m-%d),End=$(date -u +%Y-%m-%d)

# Get AWS's recommendation
aws ce get-savings-plans-purchase-recommendation \
  --savings-plans-type COMPUTE_SP \
  --term-in-years ONE_YEAR \
  --payment-option NO_UPFRONT \
  --lookback-period-in-days SIXTY_DAYS

Cost allocation tagging

The activation trap is the single biggest blocker to per-team / per-env cost attribution.

The flow:

1. Tag resources with team, env, cost-center, etc. — usually via Terraform default_tags on the provider.
2. Activate each tag in Billing → Cost Allocation Tags → User-Defined Cost Allocation Tags. Without this step, the tag is invisible to Cost Explorer and CUR.
3. Wait 24–48 hours for the tag to start appearing in cost data.

# Activate a tag via CLI (uses the Cost Explorer API)
aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status TagKey=team,Status=Active TagKey=env,Status=Active

Critical retroactivity rule: activation applies forward only. Costs incurred BEFORE the activation date are not retroactively tagged. The "before" period will continue to show (no tag) for that dimension permanently.

Residual (no tag) after activation — some costs never carry tags regardless:

• AWS Support charges
• Marketplace charges
• Data transfer between regions (untagged)
• Some AWS-managed resources (default VPC NAT gateway, AWS-managed KMS keys)
• Untagged historical data (per the retroactivity rule)

A residual of ~5–15% (no tag) is normal. >50% means either tags aren't activated or many resources are untagged.

Forward-going enforcement:

• AWS Config rule required-tags — flags non-compliant resources.
• SCP at the Organizations level denying aws:RequestTag/team = null on resource creation.
• Terraform default_tags { tags = { team = "..." } } at the provider — automatic for everything that supports it (which is most resources, but not all).

Cost & Usage Report (CUR)

Use CUR over Cost Explorer when:

• Lookback >14 months (Cost Explorer max).
• Custom grouping not in Cost Explorer (resource_id × usage_type × tag, simultaneously).
• Building dashboards or pipelines on top of cost data.

Setup (one-time):

1. Billing → Data Exports → Create Export → Standard data export → CUR 2.0. New exports are only created via the Data Exports page; legacy CUR exports created through the older "Cost & Usage Reports" page still work but new ones must go through Data Exports.
2. Lands as Parquet in an S3 bucket, partitioned by billing_period.
3. Optionally enable Athena integration — AWS generates a CloudFormation template that creates the table + Glue crawler.

Common Athena queries:

-- Top 20 resource IDs by cost, last month
SELECT line_item_resource_id, SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE billing_period = DATE '2026-04-01'
  AND line_item_resource_id <> ''
GROUP BY line_item_resource_id
ORDER BY cost DESC
LIMIT 20;

-- Untagged cost by service
SELECT line_item_product_code, SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE billing_period = DATE '2026-04-01'
  AND (resource_tags_user_team IS NULL OR resource_tags_user_team = '')
GROUP BY line_item_product_code
ORDER BY cost DESC;

Tag columns are flattened as resource_tags_user_<tag_key> in CUR 2.0. Tag activation in Billing is still required.

Anti-patterns checklist

• ❌ Diagnosing a spike by guessing services before running Cost Explorer.
• ❌ Comparing a 7-day spike against month-to-date totals. Use equal windows.
• ❌ Caching node_modules (or similar) in S3 instead of fixing the underlying download path — wasted effort vs the real fix (VPC endpoint, cache the download cache, etc).
• ❌ Buying a Savings Plan or RI at 100% of current usage. Target 70% of steady-state.
• ❌ Buying Convertible RIs. Compute Savings Plan strictly dominates for any new commitment.
• ❌ Tagging resources without activating the tag in Billing. Tags are invisible to Cost Explorer / CUR until activated.
• ❌ Expecting tag activation to retroactively label pre-activation costs. It doesn't.
• ❌ Using NAT for routine S3 / DynamoDB egress. Gateway endpoints are free.
• ❌ CloudWatch log groups with no retention. Set retention at the Terraform level, before the producer creates the group.
• ❌ Quoting specific dollar amounts as pricing facts in skill content. State relative magnitudes; link to AWS pricing for precise current numbers.

Cross-skill notes

• For VPC endpoint setup details beyond the cost trade-off (subnet placement, security group rules, DNS resolution), see aws-vpc-networking (when built) — this skill states the rule; that skill goes deep on setup.
• For CodeBuild + VPC endpoint patterns (ECR pulls, S3 artifact buckets), see aws-codepipeline-codebuild.
• For Terraform-managed log groups with retention set at the IaC level, see terraform-workflows.
• For Lambda log retention specifically (the auto-creation trap), see aws-lambda-operations (when built).