gabrielmoreira AWS S3
Manage Amazon S3 object storage with production-grade security, lifecycle policies, replication, and access controls.
When to Use This Skill
- Creating S3 buckets with security hardening (encryption, public access block, versioning)
- Writing bucket policies to enforce HTTPS, restrict IP ranges, or grant cross-account access
- Setting up lifecycle rules to transition objects between storage classes
- Configuring cross-region replication for disaster recovery
- Generating presigned URLs for temporary access to private objects
- Setting up static website hosting or CloudFront origins
- Troubleshooting access denied errors or policy conflicts
Prerequisites
- AWS CLI v2 installed and configured
- IAM permissions:
s3:*,s3-object-lambda:*,kms:*(for SSE-KMS) - For replication: IAM role with replication permissions and destination bucket in target region
- For logging: a separate logging bucket with appropriate ACL
Create and Secure a Bucket
# Create a bucket (us-east-1 does not need LocationConstraint)
aws s3api create-bucket \
--bucket my-app-data-prod \
--region us-east-1
# Create a bucket in another region
aws s3api create-bucket \
--bucket my-app-data-dr \
--region us-west-2 \
--create-bucket-configuration LocationConstraint=us-west-2
# Block ALL public access (always do this first)
aws s3api put-public-access-block \
--bucket my-app-data-prod \
--public-access-block-configuration '{
"BlockPublicAcls": true,
"IgnorePublicAcls": true,
"BlockPublicPolicy": true,
"RestrictPublicBuckets": true
}'
# Enable versioning
aws s3api put-bucket-versioning \
--bucket my-app-data-prod \
--versioning-configuration Status=Enabled
# Enable server-side encryption with SSE-KMS
aws s3api put-bucket-encryption \
--bucket my-app-data-prod \
--server-side-encryption-configuration '{
"Rules": [{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "aws:kms",
"KMSMasterKeyID": "alias/s3-key"
},
"BucketKeyEnabled": true
}]
}'
# Enable access logging
aws s3api put-bucket-logging \
--bucket my-app-data-prod \
--bucket-logging-status '{
"LoggingEnabled": {
"TargetBucket": "my-access-logs-bucket",
"TargetPrefix": "s3-logs/my-app-data-prod/"
}
}'
# Add tags
aws s3api put-bucket-tagging \
--bucket my-app-data-prod \
--tagging '{
"TagSet": [
{"Key": "Environment", "Value": "production"},
{"Key": "Team", "Value": "platform"},
{"Key": "DataClassification", "Value": "confidential"}
]
}'
Bucket Policies
# Apply a bucket policy (enforce HTTPS and restrict to VPC endpoint)
aws s3api put-bucket-policy \
--bucket my-app-data-prod \
--policy '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyInsecureTransport",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::my-app-data-prod",
"arn:aws:s3:::my-app-data-prod/*"
],
"Condition": {
"Bool": {"aws:SecureTransport": "false"}
}
},
{
"Sid": "RestrictToVPCEndpoint",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::my-app-data-prod",
"arn:aws:s3:::my-app-data-prod/*"
],
"Condition": {
"StringNotEquals": {
"aws:sourceVpce": "vpce-abc123"
}
}
}
]
}'
Cross-account access policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CrossAccountRead",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::987654321098:role/DataAnalystRole"
},
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-app-data-prod",
"arn:aws:s3:::my-app-data-prod/shared/*"
]
}
]
}
Lifecycle Rules
# Apply a comprehensive lifecycle configuration
aws s3api put-bucket-lifecycle-configuration \
--bucket my-app-data-prod \
--lifecycle-configuration '{
"Rules": [
{
"ID": "TierDownOldData",
"Status": "Enabled",
"Filter": {"Prefix": "data/"},
"Transitions": [
{"Days": 30, "StorageClass": "STANDARD_IA"},
{"Days": 90, "StorageClass": "GLACIER_IR"},
{"Days": 180, "StorageClass": "GLACIER"},
{"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
]
},
{
"ID": "ExpireLogs",
"Status": "Enabled",
"Filter": {"Prefix": "logs/"},
"Expiration": {"Days": 90},
"Transitions": [
{"Days": 7, "StorageClass": "STANDARD_IA"},
{"Days": 30, "StorageClass": "GLACIER"}
]
},
{
"ID": "CleanupOldVersions",
"Status": "Enabled",
"Filter": {"Prefix": ""},
"NoncurrentVersionTransitions": [
{"NoncurrentDays": 30, "StorageClass": "STANDARD_IA"},
{"NoncurrentDays": 90, "StorageClass": "GLACIER"}
],
"NoncurrentVersionExpiration": {"NoncurrentDays": 180}
},
{
"ID": "AbortIncompleteUploads",
"Status": "Enabled",
"Filter": {"Prefix": ""},
"AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
},
{
"ID": "ExpireDeleteMarkers",
"Status": "Enabled",
"Filter": {"Prefix": ""},
"Expiration": {"ExpiredObjectDeleteMarker": true}
}
]
}'
Cross-Region Replication
# Enable replication (requires versioning on both buckets)
aws s3api put-bucket-replication \
--bucket my-app-data-prod \
--replication-configuration '{
"Role": "arn:aws:iam::123456789012:role/S3ReplicationRole",
"Rules": [
{
"ID": "ReplicateAll",
"Status": "Enabled",
"Priority": 1,
"Filter": {"Prefix": ""},
"Destination": {
"Bucket": "arn:aws:s3:::my-app-data-dr",
"StorageClass": "STANDARD_IA",
"EncryptionConfiguration": {
"ReplicaKmsKeyID": "arn:aws:kms:us-west-2:123456789012:key/dr-key-id"
},
"Metrics": {"Status": "Enabled", "EventThreshold": {"Minutes": 15}},
"ReplicationTime": {"Status": "Enabled", "Time": {"Minutes": 15}}
},
"DeleteMarkerReplication": {"Status": "Enabled"},
"SourceSelectionCriteria": {
"SseKmsEncryptedObjects": {"Status": "Enabled"}
}
}
]
}'
# Check replication status
aws s3api head-object \
--bucket my-app-data-prod \
--key data/important-file.json \
--query "ReplicationStatus"
Presigned URLs
# Generate a presigned URL for downloading (valid 1 hour)
aws s3 presign s3://my-app-data-prod/reports/quarterly.pdf \
--expires-in 3600
# Generate a presigned URL for uploading
aws s3 presign s3://my-app-data-prod/uploads/user-file.zip \
--expires-in 3600
# Presigned URL with specific content type (using the API directly)
aws s3api generate-presigned-url \
--client-method put_object \
--params '{"Bucket":"my-app-data-prod","Key":"uploads/photo.jpg","ContentType":"image/jpeg"}' \
--expires-in 3600
Common S3 Operations
# Sync a local directory to S3
aws s3 sync ./build s3://my-app-data-prod/static/ \
--delete \
--exclude "*.tmp" \
--cache-control "max-age=31536000" \
--content-encoding "gzip"
# Copy with storage class
aws s3 cp large-archive.tar.gz s3://my-app-data-prod/archives/ \
--storage-class GLACIER_IR
# List objects with size summary
aws s3 ls s3://my-app-data-prod/ --recursive --summarize --human-readable
# Remove all objects with a prefix
aws s3 rm s3://my-app-data-prod/temp/ --recursive
# Get bucket size via CloudWatch (most efficient for large buckets)
aws cloudwatch get-metric-statistics \
--namespace AWS/S3 \
--metric-name BucketSizeBytes \
--dimensions Name=BucketName,Value=my-app-data-prod Name=StorageType,Value=StandardStorage \
--start-time "$(date -u -d '2 days ago' +%Y-%m-%dT%H:%M:%SZ)" \
--end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--period 86400 \
--statistics Average \
--output table
Terraform S3 Bucket
resource "aws_s3_bucket" "main" {
bucket = "my-app-data-prod"
tags = {
Environment = "production"
DataClassification = "confidential"
}
}
resource "aws_s3_bucket_versioning" "main" {
bucket = aws_s3_bucket.main.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_public_access_block" "main" {
bucket = aws_s3_bucket.main.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
resource "aws_s3_bucket_server_side_encryption_configuration" "main" {
bucket = aws_s3_bucket.main.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
kms_master_key_id = aws_kms_key.s3.arn
}
bucket_key_enabled = true
}
}
resource "aws_s3_bucket_lifecycle_configuration" "main" {
bucket = aws_s3_bucket.main.id
rule {
id = "tier-down"
status = "Enabled"
transition {
days = 30
storage_class = "STANDARD_IA"
}
transition {
days = 90
storage_class = "GLACIER"
}
noncurrent_version_transition {
noncurrent_days = 30
storage_class = "GLACIER"
}
noncurrent_version_expiration {
noncurrent_days = 180
}
abort_incomplete_multipart_upload {
days_after_initiation = 7
}
}
}
resource "aws_s3_bucket_policy" "enforce_https" {
bucket = aws_s3_bucket.main.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Sid = "DenyInsecureTransport"
Effect = "Deny"
Principal = "*"
Action = "s3:*"
Resource = [
aws_s3_bucket.main.arn,
"${aws_s3_bucket.main.arn}/*"
]
Condition = {
Bool = { "aws:SecureTransport" = "false" }
}
}]
})
}
Troubleshooting
| Problem | Cause | Fix |
|---|---|---|
| Access Denied on GetObject | Bucket policy or IAM denies access | Check bucket policy, IAM policy, and public access block |
| Access Denied on PutObject | Missing encryption header when required | Add SSE header; check bucket policy encryption conditions |
| 403 on presigned URL | URL expired or wrong region | Regenerate; ensure region matches bucket region |
| Replication not working | Versioning disabled on source or dest | Enable versioning on both buckets |
| Lifecycle not transitioning | Rule filter does not match objects | Verify prefix and tag filters; check rule status |
| Bucket delete fails | Bucket not empty or has versioned objects | Delete all objects and versions first; disable versioning |
| Slow uploads for large files | Single-part upload | Use aws s3 cp (auto multipart) or set multipart threshold |
| Cross-account access denied | Both bucket policy AND IAM policy needed | Grant in bucket policy and in caller's IAM policy |
| Object Lock prevents deletion | Governance or compliance mode active | Use governance bypass (with permission) or wait for retention |
Related Skills
- aws-iam - Bucket and object access policies
- aws-vpc - VPC endpoints for private S3 access
- aws-cost-optimization - Storage class optimization
- terraform-aws - IaC deployment for S3
- cloudformation - AWS-native S3 templates