AWS S3

S3 (Simple Storage Service)

Configure durable, scalable object storage with versioning, lifecycle management, and encryption. S3 provides unlimited, cost-effective storage for any data type with sophisticated access controls.

TL;DR Checklist

• Enable versioning for data protection and rollback
• Configure lifecycle policies to transition old objects to cheaper storage classes
• Enable encryption at rest (SSE-S3, SSE-KMS, or client-side)
• Implement bucket policies with principle of least privilege
• Enable block public access for all buckets by default
• Enable MFA delete for critical buckets
• Configure access logging to CloudWatch or another bucket
• Use S3 Object Lock for compliance requirements
• Enable CloudTrail logging for API audit trail
• Configure intelligent tiering for automatic cost optimization

When to Use

Use S3 when:

• Storing large files or unstructured data (images, videos, documents, backups)
• Requiring highly durable, geographically distributed storage (11 9's of durability)
• Hosting static websites or serving content via CloudFront
• Storing application backups, database snapshots, or disaster recovery data
• Building data lakes or data warehouses
• Needing versioning and point-in-time recovery
• Sharing objects with fine-grained access control

When NOT to Use

Avoid S3 when:

• Requiring microsecond latency (use ElastiCache or DynamoDB instead)
• Building highly structured relational databases (use RDS instead)
• Needing file system semantics with POSIX permissions (use EFS instead)
• Working with streaming data requiring low latency (use Kinesis instead)

Purpose and Use Cases

Primary Purpose: Provide infinitely scalable, durable object storage with automatic data protection, lifecycle management, and sophisticated access controls.

Common Use Cases:

1. Backup and Disaster Recovery — Store database backups, EC2 snapshots, and recovery data
2. Data Lake — Centralized repository for structured and unstructured data
3. Static Website Hosting — Host HTML, CSS, JavaScript with CloudFront integration
4. Log Storage — Aggregate application, security, and access logs
5. Content Distribution — Source for CloudFront CDN
6. Data Archival — Long-term storage with Glacier for compliance
7. Application Data — User uploads, generated reports, processed outputs

Architecture Design Patterns

Pattern 1: Versioned Bucket with Lifecycle Management

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  DataBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-data-bucket-unique-name
      VersioningConfiguration:
        Status: Enabled
      LifecycleConfiguration:
        Rules:
          # Transition to Intelligent-Tiering after 30 days
          - Id: TransitionToIntelligentTiering
            Status: Enabled
            Transitions:
              - TransitionInDays: 30
                StorageClass: INTELLIGENT_TIERING
          # Transition to Glacier after 90 days
          - Id: TransitionToGlacier
            Status: Enabled
            Transitions:
              - TransitionInDays: 90
                StorageClass: GLACIER
          # Expire (delete) after 365 days
          - Id: ExpireOldVersions
            Status: Enabled
            ExpirationInDays: 365
            NoncurrentVersionExpirationInDays: 90
          # Delete incomplete multipart uploads after 7 days
          - Id: CleanupIncompleteUploads
            Status: Enabled
            AbortIncompleteMultipartUpload:
              DaysAfterInitiation: 7
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
            BucketKeyEnabled: true
      LoggingConfiguration:
        DestinationBucketName: !Ref LogBucket
        LogFilePrefix: data-bucket-logs/
      Tags:
        - Key: Environment
          Value: production
        - Key: DataClassification
          Value: sensitive

  LogBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-data-bucket-logs-unique-name
      AccessControl: LogDeliveryWrite
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true

  DataBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref DataBucket
      PolicyText:
        Version: '2012-10-17'
        Statement:
          # Deny unencrypted uploads
          - Sid: DenyUnencryptedObjectUploads
            Effect: Deny
            Principal: '*'
            Action: s3:PutObject
            Resource: !Sub '${DataBucket.Arn}/*'
            Condition:
              StringNotEquals:
                s3:x-amz-server-side-encryption: AES256
          # Deny insecure transport
          - Sid: DenyInsecureTransport
            Effect: Deny
            Principal: '*'
            Action: s3:*
            Resource:
              - !GetAtt DataBucket.Arn
              - !Sub '${DataBucket.Arn}/*'
            Condition:
              Bool:
                aws:SecureTransport: 'false'
          # Allow specific IAM role to access
          - Sid: AllowApplicationAccess
            Effect: Allow
            Principal:
              AWS: arn:aws:iam::123456789012:role/ApplicationRole
            Action:
              - s3:GetObject
              - s3:PutObject
              - s3:DeleteObject
            Resource: !Sub '${DataBucket.Arn}/*'

  BucketNotification:
    Type: AWS::S3::Bucket
    Properties:
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: s3:ObjectCreated:*
            Function: arn:aws:lambda:us-east-1:123456789012:function/ProcessS3Upload
            Filter:
              Key:
                FilterRules:
                  - Name: prefix
                    Value: uploads/
                  - Name: suffix
                    Value: .jpg

Key Elements:

• Versioning enabled for data protection
• Lifecycle rules transition objects through storage classes
• Intelligent-Tiering automatically optimizes costs
• Glacier for long-term retention
• Complete public access blocking
• Server-side encryption with bucket keys
• Access logging to separate bucket
• Bucket policy enforces encrypted transport and uploads
• Event notifications trigger Lambda processing

Pattern 2: Static Website Hosting with CloudFront

Resources:
  WebsiteBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-website-bucket-unique-name
      WebsiteConfiguration:
        IndexDocument: index.html
        ErrorDocument: 404.html
      PublicAccessBlockConfiguration:
        BlockPublicAcls: false
        BlockPublicPolicy: false
        IgnorePublicAcls: false
        RestrictPublicBuckets: false

  WebsiteBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref WebsiteBucket
      PolicyText:
        Version: '2012-10-17'
        Statement:
          - Sid: PublicReadGetObject
            Effect: Allow
            Principal: '*'
            Action: s3:GetObject
            Resource: !Sub '${WebsiteBucket.Arn}/*'

  CloudFrontDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Enabled: true
        DefaultRootObject: index.html
        Origins:
          - Id: S3Origin
            DomainName: !GetAtt WebsiteBucket.RegionalDomainName
            S3OriginConfig:
              OriginAccessIdentity: !Sub 'origin-access-identity/cloudfront/${CloudFrontOAI}'
        DefaultCacheBehavior:
          AllowedMethods:
            - GET
            - HEAD
          CachedMethods:
            - GET
            - HEAD
          TargetOriginId: S3Origin
          ViewerProtocolPolicy: redirect-to-https
          ForwardedValues:
            QueryString: false
            Cookies:
              Forward: none
          Compress: true
          DefaultTTL: 86400
          MaxTTL: 31536000
        HttpVersion: http2and3

  CloudFrontOAI:
    Type: AWS::CloudFront::CloudFrontOriginAccessIdentity
    Properties:
      CloudFrontOriginAccessIdentityConfig:
        Comment: OAI for S3 website bucket

Key Elements:

• S3 bucket configured as static website
• CloudFront Origin Access Identity restricts direct bucket access
• Only CloudFront can read objects (OAI-based access)
• HTTP/2 and HTTP/3 support for performance
• Compression enabled for text content
• Long TTL for static assets

Pattern 3: Multi-Region Replication with Compliance Lock

Resources:
  PrimaryBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-primary-bucket-unique-name
      VersioningConfiguration:
        Status: Enabled
      ReplicationConfiguration:
        Role: !GetAtt ReplicationRole.Arn
        Rules:
          - Id: ReplicateAll
            Status: Enabled
            Priority: 1
            DeleteMarkerReplication:
              Status: Enabled
            Filter:
              Prefix: ''
            Destination:
              Bucket: !GetAtt ReplicaBucket.Arn
              ReplicationTime:
                Status: Enabled
                Time:
                  Minutes: 15
              Metrics:
                Status: Enabled
                EventThreshold:
                  Minutes: 15
              StorageClass: STANDARD_IA

  ReplicaBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-replica-bucket-unique-name
      VersioningConfiguration:
        Status: Enabled
      ObjectLockEnabled: true
      ObjectLockConfiguration:
        ObjectLockEnabled: Enabled
        Rule:
          DefaultRetention:
            Mode: GOVERNANCE
            Years: 7

  ReplicationRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: s3.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: S3Replication
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetReplicationConfiguration
                  - s3:ListBucket
                Resource: !GetAtt PrimaryBucket.Arn
              - Effect: Allow
                Action:
                  - s3:GetObjectVersionForReplication
                  - s3:GetObjectVersionAcl
                Resource: !Sub '${PrimaryBucket.Arn}/*'
              - Effect: Allow
                Action:
                  - s3:ReplicateObject
                  - s3:ReplicateDelete
                Resource: !Sub '${ReplicaBucket.Arn}/*'

Key Elements:

• Multi-region replication with RTC (Replication Time Control)
• Object Lock for compliance (immutable backups)
• GOVERNANCE mode allows authorized users to modify lock
• Metrics track replication status
• Replica bucket in different region for disaster recovery

Integration Approaches

1. Integration with CloudFront

S3 + CloudFront provides:

• Global content distribution with edge caching
• Reduced S3 API costs (CloudFront caches responses)
• DDoS protection via CloudFront
• Origin Access Identity for secure bucket access

2. Integration with Lambda

S3 events trigger Lambda functions for:

• Image resizing and transformation
• Document processing (PDF conversion, OCR)
• Log analysis and alerts
• Data transformation pipelines

3. Integration with Athena and Glue

Query S3 data directly via SQL:

• Athena for ad-hoc SQL queries on CSV, JSON, Parquet
• Glue Crawler to auto-discover schema
• No data movement required (query in-place)

4. Integration with CloudTrail

Enable CloudTrail for S3 API auditing:

• Track all API calls (GetObject, PutObject, DeleteObject)
• Compliance and forensics investigations
• Unauthorized access detection

5. Integration with IAM and KMS

Fine-grained access control:

• IAM policies for user/role permissions
• Bucket policies for public/cross-account access
• KMS encryption with customer-managed keys
• Resource-based policies for least privilege

Common Pitfalls

❌ Pitfall 1: Buckets Not Versioned

Problem: Accidental deletions or overwrites are permanent; no rollback possible.

Solution:

• Enable versioning on all production buckets
• Combine with lifecycle rules to manage version storage costs
• Use Object Lock for compliance requirements

❌ Pitfall 2: No Lifecycle Management

Problem: All data stored in S3 Standard costs $0.023/GB/month; old data accumulates.

Solution:

• Transition to Intelligent-Tiering after 30 days (automatic optimization)
• Move to Glacier/Deep Archive after 90 days for long-term retention
• Set expiration policies for temporary data
• Regular cost analysis via S3 Storage Lens

❌ Pitfall 3: Public Buckets Exposing Sensitive Data

Problem: Bucket misconfiguration exposes credentials, keys, or PII.

Solution:

• Enable block public access on all buckets by default
• Use bucket policies to restrict access
• Never use public ACLs
• Regular security audits via S3 Block Public Access

❌ Pitfall 4: No Encryption

Problem: Data at rest is unencrypted; fails compliance requirements.

Solution:

• Enable default encryption (SSE-S3 or SSE-KMS)
• Use bucket policies to deny unencrypted uploads
• Use customer-managed KMS keys for sensitive data
• Enable encryption for snapshots and replicas

❌ Pitfall 5: Not Monitoring Access

Problem: Unauthorized access goes undetected; compliance violations.

Solution:

• Enable access logging to CloudWatch
• Configure S3 Storage Lens for analytics
• Enable CloudTrail for API audit trail
• Set up CloudWatch alarms for suspicious patterns

❌ Pitfall 6: Missing Multi-Region Replication

Problem: Regional outage loses all data; no disaster recovery.

Solution:

• Enable S3 replication to secondary region for critical data
• Use Object Lock in replica bucket for immutability
• Test failover procedures regularly

Best Practices Summary

Related Skills

Core Workflow

1. Assess Requirements — Understand the use case, scale, integration needs, and existing infrastructure. Checkpoint: Document requirements, constraints, and success criteria.

2. Design Architecture — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. Checkpoint: Verify the architecture addresses all requirements and follows CNCF conventions.

3. Implement & Configure — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. Checkpoint: Validate all YAML against schema and test in a staging environment.

4. Deploy & Monitor — Apply manifests to the cluster, verify component health, and confirm observability is working. Checkpoint: Confirm all pods/services are running, probes passing, and metrics/alerts configured.

Constraints

MUST DO

• Include at least one complete working YAML manifest example
• Note when content is auto-generated vs. manually verified
• Reference relevant CNCF project documentation

MUST NOT DO

• Deploy manifests without testing in a staging environment first
• Use deprecated API versions (e.g., apps/v1beta1)
• Omit resource limits and requests in Kubernetes manifests