Managed Flink Diagnostics

Amazon Managed Service for Apache Flink Diagnostics

When to use

Any Managed Flink investigation — application creation, startup, Flink job failures, checkpointing, source/sink connectors, scaling, parallelism, state management, watermarks, VPC connectivity, IAM permissions, or Studio notebooks.

Investigation workflow

Step 1 — Collect and triage

aws kinesisanalyticsv2 list-applications --query 'ApplicationSummaries[*].{Name:ApplicationName,Status:ApplicationStatus,RuntimeEnvironment:RuntimeEnvironment}'
aws kinesisanalyticsv2 describe-application --application-name <app-name>
aws cloudwatch get-metric-statistics --namespace AWS/KinesisAnalytics --metric-name downtime --dimensions Name=Application,Value=<app-name> --start-time <start> --end-time <end> --period 300 --statistics Maximum

Step 2 — Domain deep dive

aws kinesisanalyticsv2 describe-application --application-name <app-name> --include-additional-details
aws logs filter-log-events --log-group-name /aws/kinesis-analytics/<app-name> --start-time <epoch-ms> --limit 100
aws cloudwatch get-metric-statistics --namespace AWS/KinesisAnalytics --metric-name lastCheckpointDuration --dimensions Name=Application,Value=<app-name> --start-time <start> --end-time <end> --period 300 --statistics Average

Step 3 — Detailed investigation

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=kinesisanalytics.amazonaws.com --max-results 20
aws kinesisanalyticsv2 describe-application-snapshot --application-name <app-name> --snapshot-name <snapshot>
aws cloudwatch get-metric-statistics --namespace AWS/KinesisAnalytics --metric-name numberOfFailedCheckpoints --dimensions Name=Application,Value=<app-name> --start-time <start> --end-time <end> --period 300 --statistics Sum

Read references/guardrails.md before concluding on any Managed Flink issue.

Tool quick reference

Gotchas: Amazon Managed Service for Apache Flink

• The service was renamed from Kinesis Data Analytics to Managed Service for Apache Flink. The API namespace is still kinesisanalyticsv2.
• Applications have TWO runtime environments: FLINK-1_15, FLINK-1_18, etc. for code-based, and ZEPPELIN-FLINK-3_0 for Studio notebooks.
• Checkpointing is critical for fault tolerance. Failed checkpoints mean the application cannot recover from failures without data loss.
• Parallelism and ParallelismPerKPU control scaling. Total parallelism = Parallelism × ParallelismPerKPU. Incorrect values cause resource waste or bottlenecks.
• VPC-connected applications cannot access the internet by default. They need NAT Gateway for internet access and VPC endpoints for AWS services.
• Application snapshots capture Flink state. They are required for safe updates and rollbacks. Always snapshot before making changes.
• Studio notebooks (Zeppelin) have different resource limits and capabilities than code-based applications.

Anti-hallucination rules

1. Always cite specific application names, CloudWatch metrics, or log entries as evidence.
2. Kinesis Data Analytics v2 API and Managed Flink are the same service. Never treat them as separate.
3. Checkpointing and savepointing are different mechanisms. Never conflate them.
4. Parallelism and ParallelismPerKPU multiply together. Never confuse individual settings with total parallelism.
5. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

14 runbooks