Amazon S3

amazon-s3 Best Practices

Amazon S3 is the backbone of object storage in AWS. Adhering to these guidelines ensures your S3 implementations are secure, performant, cost-effective, and maintainable.

1. Security First

Security is paramount. Always assume your S3 buckets contain sensitive data and configure them defensively.

1.1. Encryption at Rest

Always encrypt objects at rest. Default to AWS-managed keys (SSE-S3) or KMS (SSE-KMS) for stronger control.

❌ BAD: Unencrypted objects or relying on client-side encryption without server-side enforcement.

import boto3

s3 = boto3.client('s3')
s3.put_object(Bucket='my-unencrypted-bucket', Key='data.txt', Body=b'sensitive data')
# Data is stored unencrypted by default if bucket policy doesn't enforce it.

✅ GOOD: Enforce server-side encryption (SSE-S3 or SSE-KMS).

import boto3

s3 = boto3.client('s3')
# Option 1: SSE-S3 (AWS-managed keys)
s3.put_object(
    Bucket='my-secure-bucket',
    Key='data.txt',
    Body=b'sensitive data',
    ServerSideEncryption='AES256' # SSE-S3
)

# Option 2: SSE-KMS (Customer Master Key)
# Ensure 'my-kms-key-id' exists and bucket has permissions.
s3.put_object(
    Bucket='my-secure-bucket',
    Key='more-data.txt',
    Body=b'more sensitive data',
    ServerSideEncryption='aws:kms',
    SSEKMSKeyId='arn:aws:kms:us-east-1:123456789012:key/my-kms-key-id'
)

Note on SSE-C: As of April 2026, SSE-C is disabled for new buckets by default. Only enable it via PutBucketEncryption API after bucket creation if explicitly required for legacy or specific compliance needs. Prefer SSE-S3 or SSE-KMS.

1.2. Access Control: Policies over ACLs

Manage access exclusively through IAM and Bucket Policies. Disable ACLs for simplified, auditable permissions.

❌ BAD: Relying on ACLs for granular object access.

# Avoid this pattern. ACLs are disabled by default for new buckets.
s3.put_object_acl(
    Bucket='my-bucket',
    Key='object.txt',
    GrantRead='id="some-aws-account-id"'
)

✅ GOOD: Disable ACLs and use IAM/Bucket Policies.

# Ensure S3 Object Ownership is set to "Bucket owner enforced" to disable ACLs.
# This is the default for new buckets.
# Example Bucket Policy (IaC recommended):
# This policy grants read-only access to a specific IAM role for a specific prefix.
bucket_policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/MyReadOnlyRole"
            },
            "Action": ["s3:GetObject"],
            "Resource": ["arn:aws:s3:::my-secure-bucket/data/*"]
        }
    ]
}
# Apply this via IaC (e.g., Terraform, CloudFormation) or boto3.put_bucket_policy

1.3. Block Public Access

Always enable S3 Block Public Access at the account or organization level. Buckets should never be public unless explicitly required for static website hosting (and even then, use CloudFront with OAC).

✅ GOOD: Enable S3 Block Public Access.

# This is typically configured via AWS Console, AWS CLI, or IaC at the account level.
# Example for a specific bucket (via boto3, but IaC is preferred):
s3.put_public_access_block(
    Bucket='my-secure-bucket',
    PublicAccessBlockConfiguration={
        'BlockPublicAcls': True,
        'IgnorePublicAcls': True,
        'BlockPublicPolicy': True,
        'RestrictPublicBuckets': True
    }
)

2. Code Organization & Data Governance

Structured naming, versioning, and lifecycle rules are crucial for manageability and cost control.

2.1. Bucket & Object Naming

Adopt clear, hierarchical naming conventions, especially for data lakes. This simplifies governance and policy application.

❌ BAD: Generic or flat naming.

my-app-bucket/customerdata_20230101.csv
my-app-bucket/logs_server1_access.log

✅ GOOD: Hierarchical, environment-aware naming.

# Bucket name (globally unique, often includes org/env/purpose)
# e.g., myorg-prod-datalake-raw-us-east-1

# Object keys (within the bucket)
# For data lakes:
# <environment>/<layer>/<source>/<dataset>/<year>/<month>/<day>/<file>
# e.g., prod/raw/ecommerce/orders/2025/01/15/order_batch_1.json
# e.g., dev/curated/analytics/user_sessions/2025/01/15/session_summary.parquet

# For application assets:
# <environment>/<application>/<asset_type>/<id>/<file>
# e.g., prod/webapp/images/user-123/profile.jpg
# e.g., dev/backend/configs/service-a/config.yaml

2.2. Versioning & Tagging

Enable versioning on all critical buckets. Use object tags for cost allocation, lifecycle management, and audit trails.

❌ BAD: No versioning, untagged objects.

# Accidental deletion or overwrite means data loss.
s3.delete_object(Bucket='my-bucket', Key='important-report.csv')
# Oops, gone forever.

✅ GOOD: Enable versioning and tag objects.

# Terraform example for bucket versioning:
# resource "aws_s3_bucket" "my_bucket" {
#   bucket = "myorg-prod-data"
#   versioning {
#     enabled = true
#   }
#   # ... other configurations
# }

# Python example for tagging an object:
s3.put_object(
    Bucket='myorg-prod-data',
    Key='reports/monthly/sales-2025-01.csv',
    Body=b'sales data',
    Tagging='department=finance&project=sales-reporting'
)

2.3. Lifecycle Management

Automate object transitions to cheaper storage classes (e.g., Glacier, Intelligent-Tiering) or deletion based on access patterns. Use S3 Inventory to gain insights.

✅ GOOD: Define lifecycle rules.

# Terraform example for a lifecycle rule:
# resource "aws_s3_bucket_lifecycle_configuration" "my_bucket_lifecycle" {
#   bucket = aws_s3_bucket.my_bucket.id
#
#   rule {
#     id     = "archive_old_logs"
#     status = "Enabled"
#
#     filter {
#       prefix = "logs/"
#     }
#
#     transition {
#       days          = 30
#       storage_class = "STANDARD_IA" # Infrequent Access
#     }
#     transition {
#       days          = 90
#       storage_class = "GLACIER_FLEXIBLE_RETRIEVAL"
#     }
#     expiration {
#       days = 365
#     }
#   }
#   # ... other rules
# }

3. Performance Optimization

Design your S3 interactions for maximum throughput and minimal latency.

3.1. Multipart Uploads

For files larger than 100MB, always use multipart uploads. This improves upload speed, resilience, and allows for parallelization.

❌ BAD: Uploading large files in a single PUT operation.

# This can be slow and prone to failure for large files.
s3.put_object(Bucket='my-bucket', Key='large-video.mp4', Body=open('large-video.mp4', 'rb'))

✅ GOOD: Use transfer_manager or upload_fileobj for multipart uploads.

import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client('s3')
config = TransferConfig(multipart_threshold=1024 * 25, max_concurrency=10, multipart_chunksize=1024 * 25) # 25MB parts

with open('large-video.mp4', 'rb') as data:
    s3.upload_fileobj(data, 'my-bucket', 'large-video.mp4', Config=config)

3.2. Request Rate & Prefixes

Distribute high-throughput workloads across multiple S3 prefixes. S3 scales performance based on prefixes. Aim for at least 3-4 x 10^4 requests per second per prefix.

❌ BAD: Concentrating all high-volume reads/writes on a single prefix.

my-bucket/logs/all_access_logs_YYYY-MM-DD.log # Single hot prefix

✅ GOOD: Spread workloads across multiple prefixes.

my-bucket/logs/webserver1/access_logs_YYYY-MM-DD.log
my-bucket/logs/webserver2/access_logs_YYYY-MM-DD.log
my-bucket/logs/appserver1/access_logs_YYYY-MM-DD.log
# Or use hash-based prefixes for even distribution:
# my-bucket/data/hash_prefix_A/object_id.json
# my-bucket/data/hash_prefix_B/object_id.json

4. Infrastructure as Code (IaC) with Terraform

When using S3 as a Terraform backend, follow these critical best practices.

❌ BAD: Using local state or deprecated DynamoDB locking.

# backend "s3" {
#   bucket = "myorg-tf-states"
#   key    = "myapp/prod.tfstate"
#   region = "us-east-1"
#   # Missing versioning, encryption, and using deprecated DynamoDB for locking
#   dynamodb_table = "terraform-locks" # DEPRECATED!
# }

✅ GOOD: Enable versioning, encryption, and S3 native locking.

terraform {
  backend "s3" {
    bucket         = "myorg-tf-states"
    key            = "myapp/prod.tfstate"
    region         = "us-east-1"
    encrypt        = true             # Always encrypt state file
    versioning_enabled = true         # Critical for state recovery
    use_lockfile   = true             # RECOMMENDED: S3 native locking (Terraform 1.10+)
  }
}

5. Testing Approaches

Thoroughly test your S3 interactions, especially security and data integrity.

5.1. Local Integration Testing

Use tools like LocalStack to simulate S3 locally, allowing for rapid integration testing without incurring AWS costs or affecting production resources.

✅ GOOD: Use LocalStack for local S3 emulation.

# Example with boto3 and LocalStack (ensure LocalStack is running)
s3_local = boto3.client(
    's3',
    endpoint_url='http://localhost:4566',
    aws_access_key_id='test',
    aws_secret_access_key='test',
    region_name='us-east-1'
)
s3_local.create_bucket(Bucket='test-bucket')
s3_local.put_object(Bucket='test-bucket', Key='test-object.txt', Body=b'hello')

5.2. Policy Validation

Programmatically validate IAM and bucket policies to ensure least privilege and prevent unintended access.

✅ GOOD: Use AWS IAM Policy Simulator or boto3 for validation.

# Example: Check if an IAM role can perform a specific S3 action
iam_client = boto3.client('iam')
response = iam_client.simulate_principal_policy(
    PolicySourceArn='arn:aws:iam::123456789012:role/MyReadOnlyRole',
    ActionNames=['s3:GetObject'],
    ResourceArns=['arn:aws:s3:::my-secure-bucket/data/report.csv']
)
# Assert that 'Effect' is 'Allow' for the action.