AWS Xray

AWS X-Ray Skill

Analyze AWS X-Ray traces and service maps with parallel execution and anti-hallucination guardrails.

Relationship to other AWS skills:

• aws-xray/ → X-Ray-specific analysis (traces, service maps, sampling)
• aws/ → "How to execute" (parallel patterns, throttling, output format)

CRITICAL: Parallel Execution Requirement

ALL independent operations MUST run in parallel using background jobs (&) and wait.

#!/bin/bash
export AWS_PAGER=""

for group in $groups; do
  get_group_traces "$group" &
done
wait

Helper Functions

#!/bin/bash
export AWS_PAGER=""

# Get service graph (service map)
get_service_graph() {
  local hours=${1:-1}
  local end_time start_time
  end_time=$(date -u +"%Y-%m-%dT%H:%M:%S")
  start_time=$(date -u -d "$hours hours ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-${hours}H +"%Y-%m-%dT%H:%M:%S")
  aws xray get-service-graph \
    --start-time "$start_time" --end-time "$end_time" \
    --output text \
    --query 'Services[].[Name,Type,State,SummaryStatistics.TotalCount,SummaryStatistics.FaultStatistics.TotalCount,SummaryStatistics.ErrorStatistics.TotalCount,ResponseTimeHistogram[0].Average]'
}

# Get trace summaries with filter
get_trace_summaries() {
  local filter=$1 hours=${2:-1}
  local end_time start_time
  end_time=$(date -u +"%Y-%m-%dT%H:%M:%S")
  start_time=$(date -u -d "$hours hours ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-${hours}H +"%Y-%m-%dT%H:%M:%S")
  aws xray get-trace-summaries \
    --start-time "$start_time" --end-time "$end_time" \
    --filter-expression "$filter" \
    --output text \
    --query 'TraceSummaries[].[Id,Duration,HasFault,HasError,Http.HttpStatus,ResponseTime]' | head -20
}

# List sampling rules
list_sampling_rules() {
  aws xray get-sampling-rules \
    --output text \
    --query 'SamplingRuleRecords[].[SamplingRule.RuleName,SamplingRule.Priority,SamplingRule.FixedRate,SamplingRule.ReservoirSize,SamplingRule.ServiceName,SamplingRule.HTTPMethod,SamplingRule.URLPath]'
}

# List X-Ray groups
list_groups() {
  aws xray get-groups \
    --output text \
    --query 'Groups[].[GroupName,GroupARN,FilterExpression]'
}

Common Operations

1. Service Map Overview

#!/bin/bash
export AWS_PAGER=""
END=$(date -u +"%Y-%m-%dT%H:%M:%S")
START=$(date -u -d "1 hour ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-1H +"%Y-%m-%dT%H:%M:%S")
aws xray get-service-graph \
  --start-time "$START" --end-time "$END" \
  --output text \
  --query 'Services[].[Name,Type,SummaryStatistics.TotalCount,SummaryStatistics.FaultStatistics.TotalCount,SummaryStatistics.ErrorStatistics.TotalCount]'

2. Fault Analysis (5xx Errors)

#!/bin/bash
export AWS_PAGER=""
END=$(date -u +"%Y-%m-%dT%H:%M:%S")
START=$(date -u -d "6 hours ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-6H +"%Y-%m-%dT%H:%M:%S")
aws xray get-trace-summaries \
  --start-time "$START" --end-time "$END" \
  --filter-expression 'fault = true' \
  --output text \
  --query 'TraceSummaries[].[Id,Duration,Http.HttpStatus,Http.HttpURL,ResponseTime]' | head -20

3. Latency Investigation (Slow Traces)

#!/bin/bash
export AWS_PAGER=""
END=$(date -u +"%Y-%m-%dT%H:%M:%S")
START=$(date -u -d "1 hour ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-1H +"%Y-%m-%dT%H:%M:%S")
aws xray get-trace-summaries \
  --start-time "$START" --end-time "$END" \
  --filter-expression 'responsetime > 5' \
  --output text \
  --query 'TraceSummaries[].[Id,Duration,ResponseTime,Http.HttpURL]' | sort -k3 -rn | head -10

4. Sampling Rule Review

#!/bin/bash
export AWS_PAGER=""
aws xray get-sampling-rules \
  --output text \
  --query 'SamplingRuleRecords[].[SamplingRule.RuleName,SamplingRule.Priority,SamplingRule.FixedRate,SamplingRule.ReservoirSize,SamplingRule.ServiceName,SamplingRule.HTTPMethod,SamplingRule.URLPath,SamplingRule.Version]'

5. Error Rate by Service

#!/bin/bash
export AWS_PAGER=""
END=$(date -u +"%Y-%m-%dT%H:%M:%S")
START=$(date -u -d "6 hours ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-6H +"%Y-%m-%dT%H:%M:%S")
aws xray get-service-graph \
  --start-time "$START" --end-time "$END" \
  --output text \
  --query 'Services[].[Name,Type,SummaryStatistics.TotalCount,SummaryStatistics.FaultStatistics.TotalCount,SummaryStatistics.ErrorStatistics.TotalCount,SummaryStatistics.OkCount]' \
  | awk '{total=$3; faults=$4; errors=$5; if(total>0) printf "%s\t%s\tTotal:%s\tFaultRate:%.2f%%\tErrorRate:%.2f%%\n", $1, $2, total, faults/total*100, errors/total*100}'

Anti-Hallucination Rules

1. Fault vs Error - In X-Ray, a fault is a 5xx server error. An error is a 4xx client error. Throttle is a 429 specifically. Do not conflate them.
2. Filter expression syntax - X-Ray uses its own filter expression syntax, NOT JMESPath. Use service(), annotation.key, responsetime, fault, error keywords.
3. Trace retention - X-Ray retains trace data for 30 days. Trace summaries are available for 30 days. Full trace data beyond this requires export to S3.
4. Service graph time range - Maximum time range for get-service-graph is 6 hours per call. For longer periods, make multiple calls and aggregate.
5. Sampling affects completeness - X-Ray samples traces. A 5% sampling rate means you see ~5% of actual requests. Do not report trace counts as request counts.

Output Format

Present results as a structured report:

Aws Xray Report
═══════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Counter-Rationalizations

Common Pitfalls

• Time range limits: get-trace-summaries supports up to 6 hours per call. For longer analysis, paginate across time windows.
• Group filter expressions: Groups filter traces server-side. Different groups may show different subsets of the same traces.
• Segment documents: Full trace details require batch-get-traces with specific trace IDs. get-trace-summaries is a lightweight summary.
• CloudWatch statistics syntax: Use spaces not commas: --statistics Average Maximum.
• Active tracing vs passive: Active tracing (X-Ray SDK) sends traces. Passive tracing (downstream services) only records if upstream sends trace headers.