103 Best Practices 075aeade

AWS Best Practices for Startups

Production-ready practices for serverless, cost optimization, security, and operational excellence.

Table of Contents

• Serverless Best Practices
• Cost Optimization
• Security Hardening
• Scalability Patterns
• DevOps and Reliability
• Common Pitfalls

Serverless Best Practices

Lambda Function Design

1. Keep Functions Stateless

Store state externally in DynamoDB, S3, or ElastiCache.

# BAD: Function-level state
cache = {}

def handler(event, context):
    if event['key'] in cache:
        return cache[event['key']]
    # ...

# GOOD: External state
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('cache')

def handler(event, context):
    response = table.get_item(Key={'pk': event['key']})
    if 'Item' in response:
        return response['Item']['value']
    # ...

2. Implement Idempotency

Handle retries gracefully with unique request IDs.

import boto3
import hashlib

dynamodb = boto3.resource('dynamodb')
idempotency_table = dynamodb.Table('idempotency')

def handler(event, context):
    # Generate idempotency key
    idempotency_key = hashlib.sha256(
        f"{event['orderId']}-{event['action']}".encode()
    ).hexdigest()

    # Check if already processed
    try:
        response = idempotency_table.get_item(Key={'pk': idempotency_key})
        if 'Item' in response:
            return response['Item']['result']
    except Exception:
        pass

    # Process request
    result = process_order(event)

    # Store result for idempotency
    idempotency_table.put_item(
        Item={
            'pk': idempotency_key,
            'result': result,
            'ttl': int(time.time()) + 86400  # 24h TTL
        }
    )

    return result

3. Optimize Cold Starts

# Initialize outside handler (reused across invocations)
import boto3
from aws_xray_sdk.core import patch_all

# SDK initialization happens once
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('my-table')
patch_all()

def handler(event, context):
    # Handler code uses pre-initialized resources
    return table.get_item(Key={'pk': event['id']})

Cold Start Reduction Techniques:

• Use provisioned concurrency for critical paths
• Minimize package size (use layers for dependencies)
• Choose interpreted languages (Python, Node.js) over compiled
• Avoid VPC unless necessary (adds 6-10 sec cold start)

4. Set Appropriate Timeouts

# Lambda configuration
Functions:
  ApiHandler:
    Timeout: 10  # Shorter for synchronous APIs
    MemorySize: 512

  BackgroundProcessor:
    Timeout: 300  # Longer for async processing
    MemorySize: 1024

Timeout Guidelines:

• API handlers: 10-30 seconds
• Event processors: 60-300 seconds
• Use Step Functions for >15 minute workflows

Cost Optimization

1. Right-Sizing Strategy

# Check EC2 utilization
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \
  --start-time $(date -d '7 days ago' -u +"%Y-%m-%dT%H:%M:%SZ") \
  --end-time $(date -u +"%Y-%m-%dT%H:%M:%SZ") \
  --period 3600 \
  --statistics Average

Right-Sizing Rules:

• <10% CPU average: Downsize instance

• 80% CPU average: Consider upgrade or horizontal scaling

• Review every month for the first 6 months

2. Savings Plans and Reserved Instances

# Check Savings Plans recommendations
aws cost-explorer get-savings-plans-purchase-recommendation \
  --savings-plans-type COMPUTE_SP \
  --term-in-years ONE_YEAR \
  --payment-option NO_UPFRONT \
  --lookback-period-in-days THIRTY_DAYS

3. S3 Lifecycle Policies

{
  "Rules": [
    {
      "ID": "Transition to cheaper storage",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "logs/"
      },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

4. Lambda Memory Optimization

Test different memory settings to find optimal cost/performance.

# Use AWS Lambda Power Tuning
# https://github.com/alexcasalboni/aws-lambda-power-tuning

# Example results:
# 128 MB: 2000ms, $0.000042
# 512 MB: 500ms, $0.000042
# 1024 MB: 300ms, $0.000050

# Optimal: 512 MB (same cost, 4x faster)

5. NAT Gateway Alternatives

NAT Gateway: $0.045/hour + $0.045/GB = ~$32/month + data

Alternatives:
1. VPC Endpoints: $0.01/hour = ~$7.30/month (for AWS services)
2. NAT Instance: t3.nano = ~$3.80/month (limited throughput)
3. No NAT: Use VPC endpoints + Lambda outside VPC

6. CloudWatch Log Retention

# Set retention policies to avoid unbounded growth
LogGroup:
  Type: AWS::Logs::LogGroup
  Properties:
    LogGroupName: /aws/lambda/my-function
    RetentionInDays: 14  # 7, 14, 30, 60, 90, etc.

Retention Guidelines:

• Development: 7 days
• Production non-critical: 30 days
• Production critical: 90 days
• Compliance requirements: As specified

Security Hardening

1. IAM Least Privilege

// BAD: Overly permissive
{
  "Effect": "Allow",
  "Action": "dynamodb:*",
  "Resource": "*"
}

// GOOD: Specific actions and resources
{
  "Effect": "Allow",
  "Action": [
    "dynamodb:GetItem",
    "dynamodb:PutItem",
    "dynamodb:Query"
  ],
  "Resource": [
    "arn:aws:dynamodb:us-east-1:123456789:table/users",
    "arn:aws:dynamodb:us-east-1:123456789:table/users/index/*"
  ]
}

2. Encryption Configuration

# Enable encryption everywhere
Resources:
  # DynamoDB
  Table:
    Type: AWS::DynamoDB::Table
    Properties:
      SSESpecification:
        SSEEnabled: true
        SSEType: KMS
        KMSMasterKeyId: !Ref EncryptionKey

  # S3
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: aws:kms
              KMSMasterKeyID: !Ref EncryptionKey

  # RDS
  Database:
    Type: AWS::RDS::DBInstance
    Properties:
      StorageEncrypted: true
      KmsKeyId: !Ref EncryptionKey

3. Network Isolation

# Private subnets with VPC endpoints
Resources:
  PrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      MapPublicIpOnLaunch: false

  # DynamoDB Gateway Endpoint (free)
  DynamoDBEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref VPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.dynamodb
      VpcEndpointType: Gateway
      RouteTableIds:
        - !Ref PrivateRouteTable

  # Secrets Manager Interface Endpoint
  SecretsEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref VPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.secretsmanager
      VpcEndpointType: Interface
      PrivateDnsEnabled: true

4. Secrets Management

# Never hardcode secrets
import boto3
import json

def get_secret(secret_name):
    client = boto3.client('secretsmanager')
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response['SecretString'])

# Usage
db_creds = get_secret('prod/database/credentials')
connection = connect(
    host=db_creds['host'],
    user=db_creds['username'],
    password=db_creds['password']
)

5. API Protection

# WAF + API Gateway
WebACL:
  Type: AWS::WAFv2::WebACL
  Properties:
    DefaultAction:
      Allow: {}
    Rules:
      - Name: RateLimit
        Priority: 1
        Action:
          Block: {}
        Statement:
          RateBasedStatement:
            Limit: 2000
            AggregateKeyType: IP
        VisibilityConfig:
          SampledRequestsEnabled: true
          CloudWatchMetricsEnabled: true
          MetricName: RateLimitRule

      - Name: AWSManagedRulesCommonRuleSet
        Priority: 2
        OverrideAction:
          None: {}
        Statement:
          ManagedRuleGroupStatement:
            VendorName: AWS
            Name: AWSManagedRulesCommonRuleSet

6. Audit Logging

# Enable CloudTrail for all API calls
CloudTrail:
  Type: AWS::CloudTrail::Trail
  Properties:
    IsMultiRegionTrail: true
    IsLogging: true
    S3BucketName: !Ref AuditLogsBucket
    IncludeGlobalServiceEvents: true
    EnableLogFileValidation: true
    EventSelectors:
      - ReadWriteType: All
        IncludeManagementEvents: true

Scalability Patterns

1. Horizontal vs Vertical Scaling

Horizontal (preferred):
- Add more Lambda concurrent executions
- Add more Fargate tasks
- Add more DynamoDB capacity

Vertical (when necessary):
- Increase Lambda memory
- Upgrade RDS instance
- Larger EC2 instances

2. Database Sharding

# Partition by tenant ID
def get_table_for_tenant(tenant_id):
    shard = hash(tenant_id) % NUM_SHARDS
    return f"data-shard-{shard}"

# Or use DynamoDB single-table design with partition keys
def get_partition_key(tenant_id, entity_type, entity_id):
    return f"TENANT#{tenant_id}#{entity_type}#{entity_id}"

3. Caching Layers

Edge (CloudFront):     Global, static content, TTL: hours-days
Application (Redis):   Regional, session/query cache, TTL: minutes-hours
Database (DAX):        DynamoDB-specific, TTL: minutes

# ElastiCache Redis caching pattern
import redis
import json

cache = redis.Redis(host='cache.abc123.cache.amazonaws.com', port=6379)

def get_user(user_id):
    # Check cache first
    cached = cache.get(f"user:{user_id}")
    if cached:
        return json.loads(cached)

    # Fetch from database
    user = db.get_user(user_id)

    # Cache for 5 minutes
    cache.setex(f"user:{user_id}", 300, json.dumps(user))

    return user

4. Auto-Scaling Configuration

# ECS Service Auto-scaling
AutoScalingTarget:
  Type: AWS::ApplicationAutoScaling::ScalableTarget
  Properties:
    MaxCapacity: 10
    MinCapacity: 2
    ResourceId: !Sub service/${Cluster}/${Service.Name}
    ScalableDimension: ecs:service:DesiredCount
    ServiceNamespace: ecs

ScalingPolicy:
  Type: AWS::ApplicationAutoScaling::ScalingPolicy
  Properties:
    PolicyType: TargetTrackingScaling
    TargetTrackingScalingPolicyConfiguration:
      PredefinedMetricSpecification:
        PredefinedMetricType: ECSServiceAverageCPUUtilization
      TargetValue: 70
      ScaleInCooldown: 300
      ScaleOutCooldown: 60

DevOps and Reliability

1. Infrastructure as Code

# Version control all infrastructure
git init
git add .
git commit -m "Initial infrastructure setup"

# Use separate stacks per environment
cdk deploy --context environment=dev
cdk deploy --context environment=staging
cdk deploy --context environment=production

2. Blue/Green Deployments

# CodeDeploy Blue/Green for ECS
DeploymentGroup:
  Type: AWS::CodeDeploy::DeploymentGroup
  Properties:
    DeploymentConfigName: CodeDeployDefault.ECSAllAtOnce
    DeploymentStyle:
      DeploymentType: BLUE_GREEN
      DeploymentOption: WITH_TRAFFIC_CONTROL
    BlueGreenDeploymentConfiguration:
      DeploymentReadyOption:
        ActionOnTimeout: CONTINUE_DEPLOYMENT
        WaitTimeInMinutes: 0
      TerminateBlueInstancesOnDeploymentSuccess:
        Action: TERMINATE
        TerminationWaitTimeInMinutes: 5

3. Health Checks

# Application health endpoint
from flask import Flask, jsonify
import boto3

app = Flask(__name__)

@app.route('/health')
def health():
    checks = {
        'database': check_database(),
        'cache': check_cache(),
        'external_api': check_external_api()
    }

    status = 'healthy' if all(checks.values()) else 'unhealthy'
    code = 200 if status == 'healthy' else 503

    return jsonify({'status': status, 'checks': checks}), code

def check_database():
    try:
        # Quick connectivity test
        db.execute('SELECT 1')
        return True
    except Exception:
        return False

4. Monitoring Setup

# CloudWatch Dashboard
Dashboard:
  Type: AWS::CloudWatch::Dashboard
  Properties:
    DashboardName: production-overview
    DashboardBody: |
      {
        "widgets": [
          {
            "type": "metric",
            "properties": {
              "metrics": [
                ["AWS/Lambda", "Invocations", "FunctionName", "api-handler"],
                [".", "Errors", ".", "."],
                [".", "Duration", ".", ".", {"stat": "p99"}]
              ],
              "period": 60,
              "title": "Lambda Metrics"
            }
          }
        ]
      }

# Critical Alarms
ErrorAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: high-error-rate
    MetricName: Errors
    Namespace: AWS/Lambda
    Statistic: Sum
    Period: 60
    EvaluationPeriods: 3
    Threshold: 10
    ComparisonOperator: GreaterThanThreshold
    AlarmActions:
      - !Ref AlertTopic

Common Pitfalls

Technical Debt

Security Mistakes

Performance Issues

Cost Surprises