Bottlerocket Diagnostics

Bottlerocket Diagnostics

When to use

Any Bottlerocket investigation — AMI launch failures, OS update issues, ECS agent problems, kubelet errors, admin/control container access, networking configuration, storage issues, settings API errors, bootstrap failures, or security hardening.

Investigation workflow

Step 1 — Collect and triage

aws ec2 describe-instances --filters "Name=tag:OperatingSystem,Values=Bottlerocket" --query 'Reservations[*].Instances[*].{Id:InstanceId,State:State.Name,ImageId:ImageId,LaunchTime:LaunchTime}'
aws ssm describe-instance-information --filters "Key=PlatformType,Values=Linux" --query 'InstanceInformationList[*].{Id:InstanceId,PingStatus:PingStatus,Platform:PlatformName,Version:PlatformVersion}'
aws ec2 describe-images --owners amazon --filters "Name=name,Values=bottlerocket-aws-k8s-*" --query 'Images | sort_by(@, &CreationDate) | [-5:].{Name:Name,ImageId:ImageId,Created:CreationDate}'

Step 2 — Domain deep dive

aws ssm start-session --target <instance-id> --document-name AWS-StartInteractiveCommand --parameters command="apiclient get settings"
aws ecs describe-container-instances --cluster <cluster> --container-instances <arn> --query 'containerInstances[*].{Status:status,Agent:agentConnected,Version:versionInfo}'
aws eks describe-nodegroup --cluster-name <cluster> --nodegroup-name <nodegroup> --query 'nodegroup.{Status:status,AmiType:amiType,Version:version}'

Step 3 — Detailed investigation

aws ec2 get-console-output --instance-id <instance-id> --latest
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name StatusCheckFailed --dimensions Name=InstanceId,Value=<instance-id> --start-time <start> --end-time <end> --period 300 --statistics Maximum
aws ssm start-session --target <instance-id> --document-name AWS-StartInteractiveCommand --parameters command="logdog"

Read references/guardrails.md before concluding on any Bottlerocket issue.

Tool quick reference

Gotchas: Bottlerocket

• Bottlerocket has NO package manager and NO shell by default. The admin container must be enabled for SSH-like access. The control container provides SSM access.
• Bottlerocket uses an API-driven settings model (apiclient). Configuration files are NOT directly editable — all changes go through the settings API.
• Updates use a dual-partition (A/B) scheme. The inactive partition is updated, then the system reboots into it. Rollback swaps back to the previous partition.
• ECS and EKS variants use DIFFERENT AMIs. Using the wrong variant (e.g., k8s AMI for ECS) causes agent failures.
• The admin container is disabled by default for security. Enabling it requires user data or SSM. It provides sheltie to access the host filesystem.
• Bottlerocket logs are collected via logdog command in the control container, NOT via traditional /var/log paths.
• SELinux is enforced by default. Custom configurations that bypass SELinux will fail silently.

Anti-hallucination rules

1. Always cite specific instance IDs, AMI IDs, or API responses as evidence.
2. ECS variant and EKS variant are completely different AMIs. Never suggest using one for the other.
3. Bottlerocket has NO yum, apt, or any package manager. Never suggest installing packages.
4. Settings are API-driven only. Never suggest editing config files directly on the host.
5. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

12 runbooks