Secrets Externalization

Secrets Externalization

Overview

Secrets externalization is the practice of storing sensitive data (passwords, API keys, certificates) outside of code, version control, and infrastructure definitions. Secrets are retrieved at runtime from secure services like AWS Secrets Manager or Systems Manager Parameter Store.

Hardcoding secrets is a critical security vulnerability that leads to credential leaks, compliance failures, and breaches.

flowchart TB
    subgraph "❌ Anti-Pattern"
        A1[Code with hardcoded API key]
        A2[Committed to Git]
        A3[Exposed in public repo]
        A1 --> A2 --> A3
        style A3 fill:#FFB6C6
    end

    subgraph "✅ Best Practice"
        B1[Code references secret ARN/name]
        B2[Runtime: Fetch from Secrets Manager]
        B3[Secret injected as env var]
        B4[Application uses secret]
        B1 --> B2 --> B3 --> B4
        style B4 fill:#90EE90
    end

Key Concepts

AWS Secrets Manager

Fully managed service for storing and rotating secrets:

• Automatic rotation with Lambda
• Fine-grained IAM access control
• Encryption at rest with KMS
• Versioning and audit trail
• Cross-region replication

AWS Systems Manager Parameter Store

Hierarchical storage for configuration data and secrets:

• Free tier for standard parameters
• Integration with CloudFormation and EC2
• Versioning and change notifications
• SecureString parameters encrypted with KMS

Secret Injection Methods

1. Environment variables - Injected at container/instance startup
2. File mounts - Secrets mounted as files (ECS, Kubernetes)
3. SDK calls - Application fetches secrets at runtime
4. Init containers - Fetch secrets before main container starts

Secret Lifecycle

1. Creation - Generate and store in Secrets Manager
2. Access - IAM role grants read-only access
3. Rotation - Automatic rotation with zero downtime
4. Deletion - Soft delete with recovery window

Best Practices

1. Never commit secrets to Git - Use .gitignore for local secret files
2. Use Secrets Manager for sensitive data - Passwords, API keys, certificates
3. Use Parameter Store for config - Non-sensitive configuration values
4. Enable automatic rotation - For database credentials, API keys
5. Least privilege access - Scope IAM policies to specific secret ARNs
6. Encrypt with customer-managed KMS keys - For compliance requirements
7. Audit secret access - Enable CloudTrail logging
8. Use secret versions - For safe rotation and rollback
9. Set deletion recovery window - Prevent accidental permanent deletion
10. Replicate secrets cross-region - For disaster recovery

Anti-Patterns to Avoid

❌ Hardcoding secrets in code (API_KEY = "abc123") ❌ Committing .env files with secrets to Git ❌ Storing secrets in Terraform/CDK code ❌ Using AWS root account credentials ❌ Sharing secrets via Slack or email ❌ Long-lived IAM access keys (use IAM roles instead) ❌ Storing secrets in S3 without encryption ❌ No secret rotation (stale credentials)

Example 1: Terraform - ECS Task with Secrets from Secrets Manager

This example demonstrates:

• Storing secrets in AWS Secrets Manager
• ECS task definition referencing secrets by ARN
• Secrets injected as environment variables
• IAM policy granting read-only access to specific secrets

📁 Location: Inline example (adapt for your use case)

Key Features

# Create a secret in Secrets Manager
resource "aws_secretsmanager_secret" "db_password" {
  name        = "${local.name_prefix}/db/password"
  description = "Database password for ${var.app_name}"

  recovery_window_in_days = 7  # Soft delete with 7-day recovery

  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-db-password"
  })
}

# Store the secret value (initial version)
resource "aws_secretsmanager_secret_version" "db_password" {
  secret_id     = aws_secretsmanager_secret.db_password.id
  secret_string = jsonencode({
    username = "app_user"
    password = random_password.db_password.result
    engine   = "postgres"
    host     = aws_db_instance.main.endpoint
    port     = 5432
    dbname   = var.db_name
  })
}

# Generate random password (not stored in state plaintext)
resource "random_password" "db_password" {
  length  = 32
  special = true
}

# API key secret
resource "aws_secretsmanager_secret" "api_key" {
  name        = "${local.name_prefix}/api/key"
  description = "External API key for ${var.app_name}"

  kms_key_id = aws_kms_key.secrets.id  # Customer-managed KMS key

  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-api-key"
  })
}

resource "aws_secretsmanager_secret_version" "api_key" {
  secret_id     = aws_secretsmanager_secret.api_key.id
  secret_string = var.external_api_key  # Passed via tfvars or env var, never committed
}

# ECS Task Execution Role (can read secrets)
resource "aws_iam_role" "ecs_task_execution" {
  name = "${local.name_prefix}-ecs-task-execution-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "ecs-tasks.amazonaws.com"
      }
    }]
  })
}

# IAM policy to read specific secrets only
resource "aws_iam_role_policy" "ecs_secrets_access" {
  role = aws_iam_role.ecs_task_execution.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret"
        ]
        Resource = [
          aws_secretsmanager_secret.db_password.arn,
          aws_secretsmanager_secret.api_key.arn
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "kms:Decrypt",
          "kms:DescribeKey"
        ]
        Resource = [aws_kms_key.secrets.arn]
      }
    ]
  })
}

# Attach AWS managed policy for ECS
resource "aws_iam_role_policy_attachment" "ecs_task_execution" {
  role       = aws_iam_role.ecs_task_execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# ECS Task Definition with secrets
resource "aws_ecs_task_definition" "app" {
  family                   = "${local.name_prefix}-app"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 256
  memory                   = 512
  execution_role_arn       = aws_iam_role.ecs_task_execution.arn
  task_role_arn            = aws_iam_role.ecs_task.arn  # Different role for app permissions

  container_definitions = jsonencode([{
    name  = "app"
    image = "${var.ecr_repository_url}:${var.app_version}"

    # Secrets injected as environment variables
    secrets = [
      {
        name      = "DB_PASSWORD"
        valueFrom = "${aws_secretsmanager_secret.db_password.arn}:password::"
      },
      {
        name      = "DB_USERNAME"
        valueFrom = "${aws_secretsmanager_secret.db_password.arn}:username::"
      },
      {
        name      = "DB_HOST"
        valueFrom = "${aws_secretsmanager_secret.db_password.arn}:host::"
      },
      {
        name      = "EXTERNAL_API_KEY"
        valueFrom = aws_secretsmanager_secret.api_key.arn
      }
    ]

    # Non-sensitive environment variables
    environment = [
      {
        name  = "APP_ENV"
        value = var.environment
      },
      {
        name  = "LOG_LEVEL"
        value = "info"
      }
    ]

    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = aws_cloudwatch_log_group.app.name
        "awslogs-region"        = var.aws_region
        "awslogs-stream-prefix" = "app"
      }
    }
  }])
}

# KMS key for encrypting secrets
resource "aws_kms_key" "secrets" {
  description             = "KMS key for encrypting secrets"
  deletion_window_in_days = 10
  enable_key_rotation     = true

  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-secrets-key"
  })
}

resource "aws_kms_alias" "secrets" {
  name          = "alias/${local.name_prefix}-secrets"
  target_key_id = aws_kms_key.secrets.key_id
}

# CloudWatch Log Group
resource "aws_cloudwatch_log_group" "app" {
  name              = "/ecs/${local.name_prefix}/app"
  retention_in_days = 7

  tags = local.common_tags
}

Variable Definition (Never Commit Secrets!)

# variables.tf
variable "external_api_key" {
  description = "External API key (passed via TF_VAR_external_api_key env var)"
  type        = string
  sensitive   = true

  validation {
    condition     = length(var.external_api_key) > 0
    error_message = "external_api_key cannot be empty"
  }
}

# Pass secret via environment variable (NEVER commit to Git)
# export TF_VAR_external_api_key="your-secret-key"
# terraform apply

Example 2: CDK - Lambda Function with Secrets from Parameter Store

This example creates:

• Secrets stored in SSM Parameter Store (SecureString)
• Lambda function with IAM permission to read specific parameters
• Secrets loaded at runtime using AWS SDK

📁 Location: Inline example (adapt for your use case)

Key Features

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as ssm from 'aws-cdk-lib/aws-ssm';
import * as kms from 'aws-cdk-lib/aws-kms';
import * as iam from 'aws-cdk-lib/aws-iam';
import { Construct } from 'constructs';

export class SecretsExternalizationStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const appName = 'myapp';
    const environment = this.node.tryGetContext('environment') || 'dev';

    // KMS key for encrypting parameters
    const kmsKey = new kms.Key(this, 'ParameterKey', {
      description: `KMS key for ${appName} parameters`,
      enableKeyRotation: true,
      removalPolicy: cdk.RemovalPolicy.RETAIN,
    });

    new kms.Alias(this, 'ParameterKeyAlias', {
      aliasName: `alias/${appName}-parameters`,
      targetKey: kmsKey,
    });

    // Store database password in Parameter Store (SecureString)
    const dbPasswordParam = new ssm.StringParameter(this, 'DBPassword', {
      parameterName: `/${appName}/${environment}/db/password`,
      description: 'Database password',
      stringValue: cdk.SecretValue.unsafePlainText('placeholder').unsafeUnwrap(), // Replace after deployment
      type: ssm.ParameterType.SECURE_STRING,
      tier: ssm.ParameterTier.ADVANCED,  // For higher throughput
    });

    // Store API key
    const apiKeyParam = new ssm.StringParameter(this, 'APIKey', {
      parameterName: `/${appName}/${environment}/api/key`,
      description: 'External API key',
      stringValue: cdk.SecretValue.unsafePlainText('placeholder').unsafeUnwrap(), // Replace after deployment
      type: ssm.ParameterType.SECURE_STRING,
      tier: ssm.ParameterTier.ADVANCED,
    });

    // Store non-sensitive configuration
    const dbHostParam = new ssm.StringParameter(this, 'DBHost', {
      parameterName: `/${appName}/${environment}/db/host`,
      description: 'Database hostname',
      stringValue: 'mydb.us-east-1.rds.amazonaws.com',
      type: ssm.ParameterType.STRING,
    });

    // Lambda function code
    const lambdaCode = `
import json
import os
import boto3

ssm = boto3.client('ssm')

def handler(event, context):
    app_name = os.environ['APP_NAME']
    environment = os.environ['ENVIRONMENT']

    # Fetch secrets at runtime
    try:
        db_password = ssm.get_parameter(
            Name=f'/{app_name}/{environment}/db/password',
            WithDecryption=True
        )['Parameter']['Value']

        api_key = ssm.get_parameter(
            Name=f'/{app_name}/{environment}/api/key',
            WithDecryption=True
        )['Parameter']['Value']

        db_host = ssm.get_parameter(
            Name=f'/{app_name}/{environment}/db/host'
        )['Parameter']['Value']

        # Use secrets (never log them!)
        # db_connection = connect(host=db_host, password=db_password)
        # api_response = call_api(api_key=api_key)

        return {
            'statusCode': 200,
            'body': json.dumps({
                'message': 'Secrets loaded successfully',
                'db_host': db_host  # OK to log non-sensitive data
                # NEVER log passwords or API keys!
            })
        }
    except Exception as e:
        print(f"Error loading secrets: {str(e)}")
        return {
            'statusCode': 500,
            'body': json.dumps({'error': 'Failed to load secrets'})
        }
`;

    // Lambda function
    const fn = new lambda.Function(this, 'Function', {
      runtime: lambda.Runtime.PYTHON_3_11,
      handler: 'index.handler',
      code: lambda.Code.fromInline(lambdaCode),
      environment: {
        APP_NAME: appName,
        ENVIRONMENT: environment,
      },
      timeout: cdk.Duration.seconds(30),
    });

    // Grant Lambda permission to read specific parameters only
    dbPasswordParam.grantRead(fn);
    apiKeyParam.grantRead(fn);
    dbHostParam.grantRead(fn);

    // Grant KMS decrypt permission
    kmsKey.grantDecrypt(fn);

    // Alternative: Use Secrets Manager instead of Parameter Store
    const secretsManagerSecret = new cdk.aws_secretsmanager.Secret(this, 'DBCredentials', {
      secretName: `${appName}/${environment}/db/credentials`,
      description: 'Database credentials with automatic rotation',
      generateSecretString: {
        secretStringTemplate: JSON.stringify({ username: 'admin' }),
        generateStringKey: 'password',
        excludePunctuation: true,
        passwordLength: 32,
      },
    });

    // Grant Lambda read access to Secrets Manager
    secretsManagerSecret.grantRead(fn);

    // Outputs
    new cdk.CfnOutput(this, 'DBPasswordParamName', {
      value: dbPasswordParam.parameterName,
      description: 'Parameter Store name for DB password',
    });

    new cdk.CfnOutput(this, 'APIKeyParamName', {
      value: apiKeyParam.parameterName,
      description: 'Parameter Store name for API key',
    });

    new cdk.CfnOutput(this, 'SecretsManagerSecretArn', {
      value: secretsManagerSecret.secretArn,
      description: 'Secrets Manager ARN for DB credentials',
    });

    new cdk.CfnOutput(this, 'UpdateSecretsCommand', {
      value: [
        'aws ssm put-parameter',
        `--name "${dbPasswordParam.parameterName}"`,
        '--value "YOUR_ACTUAL_PASSWORD"',
        '--type SecureString',
        '--overwrite',
      ].join(' '),
      description: 'Command to update DB password after deployment',
    });
  }
}

Lambda with Secrets Manager SDK (Alternative)

// Lambda code using Secrets Manager
const lambdaCodeSecretsManager = `
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');

const client = new SecretsManagerClient({ region: process.env.AWS_REGION });

exports.handler = async (event) => {
    try {
        // Fetch secret from Secrets Manager
        const response = await client.send(new GetSecretValueCommand({
            SecretId: process.env.DB_SECRET_ARN
        }));

        const secret = JSON.parse(response.SecretString);

        // Use secret.username, secret.password, secret.host
        // const db = await connect({
        //     host: secret.host,
        //     user: secret.username,
        //     password: secret.password
        // });

        return {
            statusCode: 200,
            body: JSON.stringify({ message: 'Connected successfully' })
        };
    } catch (error) {
        console.error('Error fetching secret:', error);
        return {
            statusCode: 500,
            body: JSON.stringify({ error: 'Failed to fetch secret' })
        };
    }
};
`;

// Lambda with environment variable pointing to secret ARN
const fnWithSecretsManager = new lambda.Function(this, 'FunctionWithSecretsManager', {
  runtime: lambda.Runtime.NODEJS_18_X,
  handler: 'index.handler',
  code: lambda.Code.fromInline(lambdaCodeSecretsManager),
  environment: {
    DB_SECRET_ARN: secretsManagerSecret.secretArn,
  },
});

secretsManagerSecret.grantRead(fnWithSecretsManager);

Post-Deployment: Update Secrets

# Update Parameter Store secret
aws ssm put-parameter \
  --name "/myapp/prod/db/password" \
  --value "ActualSecurePassword123!" \
  --type SecureString \
  --overwrite

# Update Secrets Manager secret
aws secretsmanager update-secret \
  --secret-id myapp/prod/db/credentials \
  --secret-string '{"username":"admin","password":"ActualSecurePassword123!","host":"prod-db.amazonaws.com"}'

# Enable automatic rotation for Secrets Manager
aws secretsmanager rotate-secret \
  --secret-id myapp/prod/db/credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRotation \
  --rotation-rules AutomaticallyAfterDays=30

Validation Checklist

• No secrets committed to Git (check with git secrets or trufflehog)
• All secrets stored in Secrets Manager or Parameter Store
• IAM policies scoped to specific secret ARNs (no Resource: "*")
• Secrets encrypted with customer-managed KMS keys
• CloudTrail logging enabled for secret access
• Automatic rotation enabled for critical secrets
• .gitignore includes .env, *.pem, *.key, secrets.*
• CI/CD pipelines use OIDC (no long-lived credentials)
• Secret deletion has recovery window (7-30 days)
• Secrets replicated to DR region

Related Skills

• IAM Least Privilege - Scope access to secrets
• OIDC & IRSA Patterns - Keyless authentication
• Secrets Rotation - Automate secret rotation
• Policy as Code - Enforce no hardcoded secrets