S3 Diagnostics

S3 Diagnostics

When to use

Any S3 investigation where the console alone is insufficient — access denied analysis, bucket policy evaluation, cross-account access, replication troubleshooting, performance issues, encryption problems, or lifecycle rule debugging.

Investigation workflow

Step 1 — Collect and triage

aws s3api get-bucket-location --bucket <bucket>
aws s3api get-bucket-policy --bucket <bucket>
aws s3api get-bucket-acl --bucket <bucket>
aws s3api get-bucket-versioning --bucket <bucket>
aws s3api get-bucket-encryption --bucket <bucket>
aws s3api get-public-access-block --bucket <bucket>

Step 2 — Domain deep dive

aws s3api get-bucket-replication --bucket <bucket>
aws s3api get-bucket-lifecycle-configuration --bucket <bucket>
aws s3api get-bucket-cors --bucket <bucket>
aws s3api get-bucket-notification-configuration --bucket <bucket>
aws s3api get-bucket-logging --bucket <bucket>
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=<bucket>
aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name 4xxErrors ...

Read references/s3-guardrails.md before concluding on any S3 issue.

Tool quick reference

Gotchas: S3

• S3 access evaluation: IAM policy + bucket policy + ACL + S3 Block Public Access + VPC endpoint policy + SCP. An explicit deny in ANY of these blocks access.
• Bucket policies have a 20 KB size limit. Complex policies hit this limit.
• S3 Block Public Access has FOUR independent settings at both account and bucket level. Account-level settings override bucket-level settings.
• Object ownership: BucketOwnerEnforced disables ACLs entirely. BucketOwnerPreferred and ObjectWriter still use ACLs. New buckets default to BucketOwnerEnforced.
• Versioning cannot be disabled once enabled — only suspended. Suspended versioning still retains existing versions.
• Delete markers are NOT objects. They are zero-byte markers that make the current version appear deleted. The previous versions still exist.
• S3 Replication requires versioning on BOTH source and destination. It does NOT replicate existing objects (use S3 Batch Replication for that). Delete markers are optionally replicated.
• Lifecycle rules: transitions have minimum duration requirements (30 days to IA, 90 days to Glacier). Objects smaller than 128 KB are not transitioned to IA/Intelligent-Tiering.
• S3 request rate: 5,500 GET/HEAD and 3,500 PUT/POST/DELETE per prefix per second. Spread keys across prefixes for high throughput.
• Presigned URLs inherit the permissions of the IAM principal that created them. If the principal's permissions change, the URL may stop working.
• S3 event notifications: each event type can only have ONE destination per configuration. Duplicate event configurations cause errors.
• KMS-encrypted objects require kms:Decrypt permission on the KMS key for the reader, AND kms:GenerateDataKey for the writer.
• Cross-account access requires BOTH the source account IAM policy AND the destination bucket policy to allow access.
• S3 Transfer Acceleration uses CloudFront edge locations. It helps for long-distance uploads but adds cost.
• Multipart uploads that are not completed or aborted leave orphaned parts that incur storage costs. Use lifecycle rules to clean them up.

Storage class comparison

Anti-hallucination rules

1. Always cite specific bucket policies, IAM policies, or API responses as evidence.
2. Versioning cannot be disabled, only suspended. Never claim versioning can be turned off.
3. S3 Block Public Access account settings override bucket settings. Never ignore account-level blocks.
4. Replication does NOT replicate existing objects. Never claim enabling replication copies existing data.
5. Objects < 128 KB are NOT transitioned to IA classes. Never claim all objects transition.
6. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

34 runbooks