Amp Diagnostics

AMP Diagnostics

When to use

Any AMP investigation where the console alone is insufficient — workspace lifecycle issues, remote write ingestion failures, SigV4 authentication errors, query performance problems, alerting/ruler configuration, collector troubleshooting, or integration with Grafana and third-party tools.

Investigation workflow

Step 1 — Collect and triage

# List all workspaces and their status
aws amp list-workspaces --query 'workspaces[*].{id:workspaceId,alias:alias,status:status.statusCode}'

# Describe specific workspace
aws amp describe-workspace --workspace-id <workspace-id>

# Check CloudWatch metrics for AMP workspace
# Key metrics: ActiveSeries, IngestionRate, IngestionErrors, QueryErrors
aws cloudwatch get-metric-statistics --namespace AWS/Prometheus \
  --metric-name ActiveSeries --dimensions Name=WorkspaceId,Value=<workspace-id> \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Maximum

# Check ingestion errors
aws cloudwatch get-metric-statistics --namespace AWS/Prometheus \
  --metric-name IngestionErrors --dimensions Name=WorkspaceId,Value=<workspace-id> \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Sum

Triage returns:

• Workspace state, alias, creation timestamp
• Active series count and ingestion rate
• Ingestion and query error counts
• Workspace endpoint URLs (remote write and query)

If the workspace status is not ACTIVE, that IS the root cause domain. Don't chase ingestion or query symptoms.

Step 2 — Domain deep dive (only if needed)

# Check remote write endpoint connectivity
curl -s -o /dev/null -w "%{http_code}" --aws-sigv4 "aws:amz:us-east-1:aps" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  https://aps-workspaces.<region>.amazonaws.com/workspaces/<workspace-id>/api/v1/remote_write

# Check query endpoint
curl -s --aws-sigv4 "aws:amz:<region>:aps" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  "https://aps-workspaces.<region>.amazonaws.com/workspaces/<workspace-id>/api/v1/query?query=up"

# List alert manager configuration
aws amp describe-alert-manager-definition --workspace-id <workspace-id>

# List ruler configuration
aws amp describe-rule-groups-namespace --workspace-id <workspace-id> --name <namespace>
aws amp list-rule-groups-namespaces --workspace-id <workspace-id>

# CloudTrail for AMP API events
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=aps.amazonaws.com --max-results 20

Read references/amp-guardrails.md before concluding on any AMP issue.

Step 3 — Detailed investigation (low-confidence cases only)

# CloudTrail for detailed API call history
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=aps.amazonaws.com \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%S) --max-results 50

# VPC endpoint check (if using private connectivity)
aws ec2 describe-vpc-endpoints --filters Name=service-name,Values=com.amazonaws.<region>.aps-workspaces

# IAM role/policy verification for SigV4
aws iam simulate-principal-policy --policy-source-arn <role-arn> \
  --action-names aps:RemoteWrite aps:QueryMetrics aps:GetSeries aps:GetLabels

# Check ADOT collector logs (if running on EKS)
kubectl logs -n opentelemetry -l app=adot-collector --tail=100

# Check Prometheus server remote write status (if using Prometheus)
kubectl logs -n monitoring -l app=prometheus --tail=100 | grep -i "remote"

Tool quick reference

Gotchas: AMP

These are the mistakes commonly made during AMP troubleshooting.

• AMP is a managed Prometheus-compatible service — it is NOT self-hosted Prometheus. There is no Prometheus server to SSH into, no local TSDB, no Prometheus UI. You interact via remote write (ingestion) and query (PromQL) API endpoints only.
• Remote write to AMP requires SigV4 authentication — basic auth, bearer tokens, and other Prometheus-native auth methods do NOT work. Every request must be signed with AWS credentials using the SigV4 protocol.
• Each workspace has ingestion rate limits and active series limits. Exceeding these causes HTTP 429 (throttling) responses on the remote write endpoint. Check CloudWatch IngestionRate and ActiveSeries metrics.
• The query endpoint is separate from the remote write endpoint. Remote write URL ends in /api/v1/remote_write. Query URL uses /api/v1/query and /api/v1/query_range. Mixing them up causes 404 or method-not-allowed errors.
• AMP does NOT run a Prometheus server — there is no local storage, no Prometheus UI, no targets page, no service discovery configuration. AMP is a remote storage backend only. Use Grafana or the query API for visualization.
• Alerting in AMP uses the ruler service with SNS integration for notifications. It does NOT use a standalone Alertmanager with email/Slack/PagerDuty receivers directly — you route through SNS topics which then fan out to your notification channels.
• Metrics retention in AMP is 150 days and is NOT configurable. You cannot extend or reduce it. Plan data lifecycle accordingly.
• HA collector setups (multiple Prometheus servers or ADOT collectors scraping the same targets) send duplicate samples. AMP deduplicates based on labels, but you must ensure consistent external labels across replicas to enable proper deduplication.
• ADOT Collector and Prometheus server are both valid remote write clients, but they have different configuration formats. ADOT uses exporters.prometheusremotewrite in its YAML config. Prometheus uses remote_write in prometheus.yml. Don't mix the config syntax.
• VPC endpoints are available for private connectivity to AMP (com.amazonaws.<region>.aps-workspaces). Without a VPC endpoint, traffic goes over the public internet even from within AWS. This matters for security-sensitive workloads.
• Workspace deletion is permanent — there is no recovery, no recycle bin, no snapshot. All metrics data is lost immediately. Always confirm the workspace ID before deleting.
• Label and series cardinality is the #1 performance issue in AMP. High-cardinality labels (like request IDs, pod UIDs, or timestamps in label values) explode the active series count, leading to throttling, slow queries, and increased costs.

Anti-hallucination rules

1. Always cite specific AWS CLI output, CloudWatch metrics, CloudTrail events, or collector logs as evidence.
2. Never suggest accessing Prometheus server internals (TSDB, WAL, config reload endpoint) — AMP is fully managed with no server-side access.
3. Never recommend basic auth, bearer tokens, or API keys for AMP authentication — only SigV4 signing is supported.
4. Never claim metrics retention is configurable — it is fixed at 150 days.
5. Never suggest using Prometheus UI, targets page, or service discovery configuration — AMP has no UI; use Grafana or the query API.
6. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

26 runbooks

Runbooks are organized by failure domain. Use the appropriate runbook based on the symptom category.