Control Tower Diagnostics

Control Tower Diagnostics

When to use

Any Control Tower investigation where the console alone is insufficient — landing zone failures, account provisioning errors, guardrail compliance violations, drift detection, SCP conflicts, SSO access issues, or customization pipeline failures.

Investigation workflow

Step 1 — Collect and triage

# Get landing zone status
aws controltower list-landing-zones
aws controltower get-landing-zone --landing-zone-identifier <arn>

# Check for drift
aws controltower list-landing-zone-operations --filter statuses=FAILED

# List enabled controls (guardrails)
aws controltower list-enabled-controls --target-identifier <ou-arn>

# Check account factory provisioned products
aws servicecatalog search-provisioned-products --access-level-filter Key=Account,Value=self

# Check Organizations state
aws organizations describe-organization
aws organizations list-roots
aws organizations list-organizational-units-for-parent --parent-id <root-id>

Triage returns:

• Landing zone version and status
• Failed operations and drift indicators
• Enabled controls and compliance state
• Account provisioning status
• Organizational structure

If landing zone status is not ACTIVE, that IS the root cause domain. Don't chase downstream symptoms.

Step 2 — Domain deep dive (only if needed)

# Control (guardrail) details
aws controltower get-enabled-control --enabled-control-identifier <arn>
aws controltower list-enabled-controls --target-identifier <ou-arn>

# CloudFormation stacks (Control Tower uses StackSets)
aws cloudformation list-stack-sets --status ACTIVE
aws cloudformation describe-stack-set --stack-set-name AWSControlTowerBP-*
aws cloudformation list-stack-instances --stack-set-name <name> --filters Name=DETAILED_STATUS,Values=FAILED

# Config compliance
aws configservice describe-compliance-by-config-rule --compliance-types NON_COMPLIANT

# SCPs
aws organizations list-policies-for-target --target-id <ou-id> --filter SERVICE_CONTROL_POLICY
aws organizations describe-policy --policy-id <id>

# SSO / IAM Identity Center
aws sso-admin list-instances
aws sso-admin list-permission-sets --instance-arn <arn>

Read references/control-tower-guardrails.md before concluding on any Control Tower issue.

Step 3 — Detailed path (low-confidence cases only)

# CloudTrail for API-level events
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateManagedAccount

# CloudFormation stack events for failed provisioning
aws cloudformation describe-stack-events --stack-name <stack>

# Service Catalog provisioned product details
aws servicecatalog describe-provisioned-product --id <id>
aws servicecatalog describe-record --id <record-id>

Tool quick reference

Gotchas: Control Tower

These are the mistakes commonly made during Control Tower troubleshooting.

• Control Tower manages resources via CloudFormation StackSets. Manual changes to these stacks cause DRIFT. Never modify CT-managed stacks directly.
• Landing zone updates are NOT backward compatible. Always check the release notes before updating. Some updates require re-registering OUs.
• Account Factory uses Service Catalog under the hood. Provisioned product failures are often CloudFormation failures in the target account.
• Guardrails are now called "controls." There are three types: preventive (SCPs), detective (Config rules), and proactive (CloudFormation hooks). They behave very differently.
• Preventive controls (SCPs) block actions at the API level. They cannot be overridden by IAM policies. They apply to all principals in the account EXCEPT the management account.
• Detective controls (Config rules) only DETECT non-compliance. They do NOT prevent the action. Remediation is separate.
• The management account is NOT governed by SCPs. Never assume SCPs protect the management account.
• Control Tower creates a CloudTrail organization trail. Creating additional trails may cause duplicate log delivery and increased costs.
• The Audit account and Log Archive account are special. Moving or modifying them causes drift.
• Nested OUs are supported but controls are NOT inherited by child OUs. You must enable controls on each OU individually.
• Account email addresses must be globally unique across ALL AWS accounts. Reusing emails causes provisioning failures.
• Control Tower uses specific IAM roles (AWSControlTowerExecution, AWSControlTowerAdmin, etc.). Modifying or deleting these causes drift and operational failures.

Control types comparison

Landing zone states

Account provisioning states

Gotchas: Drift

• Drift occurs when CT-managed resources are modified outside of Control Tower.
• Common drift causes: manual SCP changes, OU moves, role modifications, stack changes.
• Drift must be resolved before you can update the landing zone or enable new controls.
• Re-registering an OU resets its CT-managed resources to the expected state.
• Account-level drift (moved account) requires moving the account back or re-registering.

Anti-hallucination rules

1. Always cite specific API responses, CloudFormation events, or Config compliance results as evidence.
2. SCPs do NOT apply to the management account. Never claim SCPs protect the management account.
3. Controls are NOT inherited by child OUs. Never claim enabling a control on a parent OU protects child OUs.
4. Detective controls detect but do NOT prevent. Never claim a detective control blocks an action.
5. Manual changes to CT-managed resources cause drift. Never recommend directly editing CT-managed CloudFormation stacks or SCPs.
6. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

34 runbooks

Runbooks are organized by failure domain. Use the appropriate runbook based on the symptom category.