# AWS Sqs > "Deploys managed message queues with SQS for asynchronous processing" decoupling services, and reliable message delivery with visibility timeout and dead-letter queues. - Skill: `paulpas/aws-sqs` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/aws-sqs` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/aws-sqs/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/aws-sqs --- # SQS (Simple Queue Service) Deploy managed message queues for asynchronous processing with guaranteed delivery, visibility timeout, and built-in dead-letter queue support. ## TL;DR Checklist - [ ] Use FIFO queue when message order is critical - [ ] Use Standard queue for high throughput when order doesn't matter - [ ] Configure visibility timeout appropriately (match processing time) - [ ] Enable message deduplication for FIFO - [ ] Set up dead-letter queue for poison pill messages - [ ] Use short polling to reduce costs (default) - [ ] Encrypt messages with KMS for sensitive data - [ ] Monitor queue depth and processing time - [ ] Set message retention appropriately (4 days default) - [ ] Use batch operations for better performance --- ## When to Use Use SQS when: - Decoupling producers and consumers - Building asynchronous processing pipelines - Implementing reliable message delivery - Handling traffic spikes with queuing - Processing messages in order (FIFO) - Retrying failed operations --- ## When NOT to Use Avoid SQS when: - Requiring real-time processing (< 1 second latency) - Need pub/sub broadcast (use SNS instead) - Strict ordering not needed but throughput critical (SNS cheaper) --- ## Purpose and Use Cases **Primary Purpose:** Provide reliable asynchronous message delivery with automatic scaling, deduplication, and failure handling. **Common Use Cases:** 1. **Async Job Processing** — Background tasks, image resizing, reports 2. **Service Decoupling** — Decouple order service from email service 3. **Rate Limiting** — Queue traffic to prevent downstream overload 4. **Retry Logic** — Automatic retries via visibility timeout 5. **Batch Processing** — Accumulate messages for batch processing 6. **Data Pipelines** — Multi-stage processing workflows --- ## Architecture Design Patterns ### Pattern 1: Standard Queue with Dead-Letter Queue ```yaml AWSTemplateFormatVersion: '2010-09-09' Resources: # Dead-Letter Queue DeadLetterQueue: Type: AWS::SQS::Queue Properties: QueueName: processing-dlq MessageRetentionPeriod: 1209600 # 14 days VisibilityTimeout: 300 KmsMasterKeyId: alias/aws/sqs # Encryption # Main Processing Queue ProcessingQueue: Type: AWS::SQS::Queue Properties: QueueName: processing-queue VisibilityTimeout: 300 # 5 minutes MessageRetentionPeriod: 345600 # 4 days ReceiveMessageWaitTimeSeconds: 10 # Long polling RedrivePolicy: deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn maxReceiveCount: 3 # Move to DLQ after 3 failures KmsMasterKeyId: alias/aws/sqs # Queue Policy QueuePolicy: Type: AWS::SQS::QueuePolicy Properties: Queues: - !Ref ProcessingQueue PolicyText: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: sns.amazonaws.com Action: sqs:SendMessage Resource: !GetAtt ProcessingQueue.Arn Condition: ArnEquals: aws:SourceArn: arn:aws:sns:*:*:* - Effect: Allow Principal: AWS: arn:aws:iam::123456789012:role/ApplicationRole Action: - sqs:SendMessage - sqs:ReceiveMessage - sqs:DeleteMessage - sqs:GetQueueAttributes Resource: !GetAtt ProcessingQueue.Arn - Effect: Deny Principal: '*' Action: sqs:* Resource: !GetAtt ProcessingQueue.Arn Condition: Bool: aws:SecureTransport: 'false' # Lambda Processor ProcessorFunction: Type: AWS::Lambda::Function Properties: FunctionName: queue-processor Runtime: python3.11 Handler: index.handler Role: !GetAtt ProcessorRole.Arn Timeout: 300 Code: ZipFile: | import json import boto3 import logging logger = logging.getLogger() def handler(event, context): sqs = boto3.client('sqs') for record in event['Records']: try: message_body = json.loads(record['body']) logger.info(f"Processing: {message_body}") # Business logic here process_message(message_body) # Delete on success sqs.delete_message( QueueUrl=record['eventSourceARN'], ReceiptHandle=record['receiptHandle'] ) except Exception as e: logger.error(f"Failed to process: {str(e)}") # On error, message stays in queue for retry raise return {'statusCode': 200} def process_message(message): # Business logic pass ProcessorRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: lambda.amazonaws.com Action: sts:AssumeRole ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Policies: - PolicyName: SQSAccess PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - sqs:ReceiveMessage - sqs:DeleteMessage - sqs:GetQueueAttributes - sqs:ChangeMessageVisibility Resource: !GetAtt ProcessingQueue.Arn - Effect: Allow Action: - kms:Decrypt Resource: !Sub 'arn:aws:kms:${AWS::Region}:${AWS::AccountId}:key/*' # Event Source Mapping EventSourceMapping: Type: AWS::Lambda::EventSourceMapping Properties: EventSourceArn: !GetAtt ProcessingQueue.Arn FunctionName: !Ref ProcessorFunction BatchSize: 10 MaximumBatchingWindowInSeconds: 5 FunctionResponseTypes: - ReportBatchItemFailures ScalingConfig: MaximumConcurrency: 10 # CloudWatch Alarms QueueDepthAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmName: sqs-queue-depth-high MetricName: ApproximateNumberOfMessagesVisible Namespace: AWS/SQS Statistic: Average Period: 300 EvaluationPeriods: 2 Threshold: 1000 ComparisonOperator: GreaterThanThreshold Dimensions: - Name: QueueName Value: !GetAtt ProcessingQueue.QueueName DLQDepthAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmName: sqs-dlq-not-empty MetricName: ApproximateNumberOfMessagesVisible Namespace: AWS/SQS Statistic: Average Period: 300 EvaluationPeriods: 1 Threshold: 1 ComparisonOperator: GreaterThanOrEqualToThreshold Dimensions: - Name: QueueName Value: !GetAtt DeadLetterQueue.QueueName ProcessingTimeAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmName: sqs-message-age-high MetricName: ApproximateAgeOfOldestMessage Namespace: AWS/SQS Statistic: Maximum Period: 300 EvaluationPeriods: 1 Threshold: 600 # 10 minutes ComparisonOperator: GreaterThanThreshold Dimensions: - Name: QueueName Value: !GetAtt ProcessingQueue.QueueName Outputs: QueueUrl: Value: !Ref ProcessingQueue Description: Queue URL QueueArn: Value: !GetAtt ProcessingQueue.Arn Description: Queue ARN DLQUrl: Value: !Ref DeadLetterQueue Description: Dead-Letter Queue URL ``` **Key Elements:** - Standard queue for high throughput - Dead-letter queue for failed messages - Long polling (10s) to reduce costs - Visibility timeout matches processing time - Message deduplication via timestamp - KMS encryption - Lambda with batch processing - Alarms for queue depth and age ### Pattern 2: FIFO Queue for Ordered Processing ```yaml Resources: # FIFO Queue (message order guaranteed) OrderQueue: Type: AWS::SQS::Queue Properties: QueueName: order-processing.fifo FifoQueue: true ContentBasedDeduplication: false # Explicit IDs preferred VisibilityTimeout: 300 MessageRetentionPeriod: 345600 ReceiveMessageWaitTimeSeconds: 10 DeduplicationScope: messageGroup # Dedup per group FifoThroughputLimit: perMessageGroupId # FIFO Dead-Letter Queue OrderDLQ: Type: AWS::SQS::Queue Properties: QueueName: order-dlq.fifo FifoQueue: true ContentBasedDeduplication: true MessageRetentionPeriod: 1209600 # Link DLQ OrderQueueWithDLQ: Type: AWS::SQS::Queue Properties: QueueName: order-queue.fifo FifoQueue: true RedrivePolicy: deadLetterTargetArn: !GetAtt OrderDLQ.Arn maxReceiveCount: 3 ``` **Key Elements:** - FIFO queue for strict ordering - MessageGroupId for partitioning (same group = same order) - ContentBasedDeduplication for automatic dedup - Deduplication scope per group - Throughput limit per message group ### Pattern 3: Batch Processing with Visibility Adjustment ```yaml Resources: # Batch Processing Lambda BatchProcessorFunction: Type: AWS::Lambda::Function Properties: FunctionName: batch-processor Runtime: python3.11 Handler: index.handler Timeout: 600 # 10 minutes for batch Code: ZipFile: | import boto3 import json from datetime import datetime, timedelta sqs = boto3.client('sqs') def handler(event, context): queue_url = event['QueueUrl'] # Receive batch of messages response = sqs.receive_message( QueueUrl=queue_url, MaxNumberOfMessages=10, WaitTimeSeconds=10 ) messages = response.get('Messages', []) try: # Process batch process_batch(messages) # Delete processed messages for msg in messages: sqs.delete_message( QueueUrl=queue_url, ReceiptHandle=msg['ReceiptHandle'] ) except Exception as e: # Extend visibility for slower processing for msg in messages: sqs.change_message_visibility( QueueUrl=queue_url, ReceiptHandle=msg['ReceiptHandle'], VisibilityTimeout=600 # +10 minutes ) raise return {'processed': len(messages)} def process_batch(messages): # Batch processing logic pass ``` **Key Elements:** - Batch receive for efficiency - Dynamic visibility adjustment - Graceful failure handling - Message deletion on success --- ## Integration Approaches ### 1. Integration with Lambda Lambda + SQS enables: - Event-driven processing - Batch message handling - Automatic scaling ### 2. Integration with SNS SNS + SQS enables: - Fan-out pattern - Persistence layer - Multiple consumers ### 3. Integration with Application Load Balancer ALB + SQS enables: - HTTP request queueing - Backend decoupling - Request buffering --- ## Common Pitfalls ### ❌ Pitfall 1: Visibility Timeout Too Short **Problem:** Messages re-delivered before processing completes; duplicates. **Solution:** - Set to match longest processing time - Use change_message_visibility if processing slower ### ❌ Pitfall 2: No Dead-Letter Queue **Problem:** Poison pill messages cause infinite retries. **Solution:** - Configure DLQ with maxReceiveCount - Monitor DLQ for failures ### ❌ Pitfall 3: Long Polling Disabled **Problem:** Empty responses waste API calls; high costs. **Solution:** - Enable long polling (WaitTimeSeconds = 10s) - Reduces API call costs by 80%+ ### ❌ Pitfall 4: Not Deleting Processed Messages **Problem:** Messages re-delivered indefinitely. **Solution:** - Always delete on successful processing - Only delete after message handle no longer valid ### ❌ Pitfall 5: FIFO for Non-Ordered Workloads **Problem:** FIFO throughput limited (300 msg/s per group). **Solution:** - Use Standard queue for high throughput - FIFO only when order matters --- ## Best Practices Summary | Category | Best Practice | |---|---| | **Queue Type** | Standard for throughput; FIFO for order | | **Visibility** | Match to processing time; adjust dynamically | | **Reliability** | Dead-letter queue; batch failure reporting | | **Performance** | Long polling; batch receives; 10+ messages | | **Monitoring** | Queue depth; message age; DLQ depth | --- ## Related Skills | Skill | Purpose | |---|---| | `cncf-aws-sns` | Fan-out to multiple queues | | `cncf-aws-lambda` | Event source for queue processing | | `cncf-aws-cloudwatch` | Queue monitoring and alarms | --- ## Core Workflow 1. **Assess Requirements** — Understand the use case, scale, integration needs, and existing infrastructure. **Checkpoint:** Document requirements, constraints, and success criteria. 2. **Design Architecture** — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. **Checkpoint:** Verify the architecture addresses all requirements and follows CNCF conventions. 3. **Implement & Configure** — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. **Checkpoint:** Validate all YAML against schema and test in a staging environment. 4. **Deploy & Monitor** — Apply manifests to the cluster, verify component health, and confirm observability is working. **Checkpoint:** Confirm all pods/services are running, probes passing, and metrics/alerts configured. --- ## Constraints ### MUST DO - Include at least one complete working YAML manifest example - Note when content is auto-generated vs. manually verified - Reference relevant CNCF project documentation ### MUST NOT DO - Deploy manifests without testing in a staging environment first - Use deprecated API versions (e.g., apps/v1beta1) - Omit resource limits and requests in Kubernetes manifests --- ## Live References > Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content. - [Primary Documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html) - [API Reference or Getting Started](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html) - [Configuration Guide](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-timing.html) - [Best Practices](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) - [Common Patterns or Tutorials](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-scheduled-concurrency-for-lambda-functions.html)