Evidently Diagnostics

Amazon CloudWatch Evidently Diagnostics

When to use

Any CloudWatch Evidently investigation — project creation, feature flag management, feature evaluations, launch configuration, experiment setup, segment configuration, metric definitions, or data export.

Investigation workflow

Step 1 — Collect and triage

aws evidently list-projects --query 'projects[*].{Name:name,Status:status,Arn:arn}'
aws evidently get-project --project <name> --query '{Name:name,Status:status,DataDelivery:dataDelivery}'
aws evidently list-features --project <name> --query 'features[*].{Name:name,Status:status,EvaluationStrategy:evaluationStrategy}'

Step 2 — Domain deep dive

aws evidently get-feature --project <name> --feature <feature-name>
aws evidently list-launches --project <name> --query 'launches[*].{Name:name,Status:status,Type:type}'
aws evidently list-experiments --project <name> --query 'experiments[*].{Name:name,Status:status,Type:type}'

Step 3 — Detailed investigation

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=evidently.amazonaws.com --max-results 20
aws evidently evaluate-feature --project <name> --feature <feature-name> --entity-id test-entity
aws evidently get-experiment-results --project <name> --experiment <experiment-name> --metric-names <metric> --treatment-names <treatment> --report-names BayesianInference

Read references/guardrails.md before concluding on any Evidently issue.

Tool quick reference

Gotchas: CloudWatch Evidently

• Evidently projects are regional. Features, launches, and experiments exist within a project in a specific region.
• Feature evaluations use either client-side (local) or server-side evaluation. Client-side requires the SDK to download rules; server-side calls the EvaluateFeature API.
• Launches control gradual feature rollouts with traffic splits. They do NOT collect metrics — use experiments for metric analysis.
• Experiments require metric definitions and a baseline variation. Results use Bayesian inference and require sufficient sample size for statistical significance.
• Segments define subsets of users for targeted launches or experiments. They use pattern matching on entity attributes.
• Data delivery to S3 or CloudWatch Logs must be configured at the project level. It is not enabled by default.
• Feature variations can be boolean, string, long, or double types. The variation type cannot be changed after feature creation.

Anti-hallucination rules

1. Always cite specific project names, feature names, or API responses as evidence.
2. Launches and experiments are different concepts. Launches control rollout; experiments measure impact. Never conflate them.
3. Feature evaluations depend on evaluation strategy (client-side vs server-side). Never assume one strategy.
4. Experiment results require statistical significance. Never claim results are conclusive without sufficient sample size.
5. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

12 runbooks