AWS Sqs

SQS (Simple Queue Service)

Deploy managed message queues for asynchronous processing with guaranteed delivery, visibility timeout, and built-in dead-letter queue support.

TL;DR Checklist

• Use FIFO queue when message order is critical
• Use Standard queue for high throughput when order doesn't matter
• Configure visibility timeout appropriately (match processing time)
• Enable message deduplication for FIFO
• Set up dead-letter queue for poison pill messages
• Use short polling to reduce costs (default)
• Encrypt messages with KMS for sensitive data
• Monitor queue depth and processing time
• Set message retention appropriately (4 days default)
• Use batch operations for better performance

When to Use

Use SQS when:

• Decoupling producers and consumers
• Building asynchronous processing pipelines
• Implementing reliable message delivery
• Handling traffic spikes with queuing
• Processing messages in order (FIFO)
• Retrying failed operations

When NOT to Use

Avoid SQS when:

• Requiring real-time processing (< 1 second latency)
• Need pub/sub broadcast (use SNS instead)
• Strict ordering not needed but throughput critical (SNS cheaper)

Purpose and Use Cases

Primary Purpose: Provide reliable asynchronous message delivery with automatic scaling, deduplication, and failure handling.

Common Use Cases:

1. Async Job Processing — Background tasks, image resizing, reports
2. Service Decoupling — Decouple order service from email service
3. Rate Limiting — Queue traffic to prevent downstream overload
4. Retry Logic — Automatic retries via visibility timeout
5. Batch Processing — Accumulate messages for batch processing
6. Data Pipelines — Multi-stage processing workflows

Architecture Design Patterns

Pattern 1: Standard Queue with Dead-Letter Queue

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  # Dead-Letter Queue
  DeadLetterQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: processing-dlq
      MessageRetentionPeriod: 1209600  # 14 days
      VisibilityTimeout: 300
      KmsMasterKeyId: alias/aws/sqs  # Encryption

  # Main Processing Queue
  ProcessingQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: processing-queue
      VisibilityTimeout: 300  # 5 minutes
      MessageRetentionPeriod: 345600  # 4 days
      ReceiveMessageWaitTimeSeconds: 10  # Long polling
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn
        maxReceiveCount: 3  # Move to DLQ after 3 failures
      KmsMasterKeyId: alias/aws/sqs

  # Queue Policy
  QueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref ProcessingQueue
      PolicyText:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: sns.amazonaws.com
            Action: sqs:SendMessage
            Resource: !GetAtt ProcessingQueue.Arn
            Condition:
              ArnEquals:
                aws:SourceArn: arn:aws:sns:*:*:*
          - Effect: Allow
            Principal:
              AWS: arn:aws:iam::123456789012:role/ApplicationRole
            Action:
              - sqs:SendMessage
              - sqs:ReceiveMessage
              - sqs:DeleteMessage
              - sqs:GetQueueAttributes
            Resource: !GetAtt ProcessingQueue.Arn
          - Effect: Deny
            Principal: '*'
            Action: sqs:*
            Resource: !GetAtt ProcessingQueue.Arn
            Condition:
              Bool:
                aws:SecureTransport: 'false'

  # Lambda Processor
  ProcessorFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: queue-processor
      Runtime: python3.11
      Handler: index.handler
      Role: !GetAtt ProcessorRole.Arn
      Timeout: 300
      Code:
        ZipFile: |
          import json
          import boto3
          import logging
          
          logger = logging.getLogger()
          
          def handler(event, context):
              sqs = boto3.client('sqs')
              
              for record in event['Records']:
                  try:
                      message_body = json.loads(record['body'])
                      logger.info(f"Processing: {message_body}")
                      
                      # Business logic here
                      process_message(message_body)
                      
                      # Delete on success
                      sqs.delete_message(
                          QueueUrl=record['eventSourceARN'],
                          ReceiptHandle=record['receiptHandle']
                      )
                      
                  except Exception as e:
                      logger.error(f"Failed to process: {str(e)}")
                      # On error, message stays in queue for retry
                      raise
              
              return {'statusCode': 200}
          
          def process_message(message):
              # Business logic
              pass

  ProcessorRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: SQSAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - sqs:ReceiveMessage
                  - sqs:DeleteMessage
                  - sqs:GetQueueAttributes
                  - sqs:ChangeMessageVisibility
                Resource: !GetAtt ProcessingQueue.Arn
              - Effect: Allow
                Action:
                  - kms:Decrypt
                Resource: !Sub 'arn:aws:kms:${AWS::Region}:${AWS::AccountId}:key/*'

  # Event Source Mapping
  EventSourceMapping:
    Type: AWS::Lambda::EventSourceMapping
    Properties:
      EventSourceArn: !GetAtt ProcessingQueue.Arn
      FunctionName: !Ref ProcessorFunction
      BatchSize: 10
      MaximumBatchingWindowInSeconds: 5
      FunctionResponseTypes:
        - ReportBatchItemFailures
      ScalingConfig:
        MaximumConcurrency: 10

  # CloudWatch Alarms
  QueueDepthAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: sqs-queue-depth-high
      MetricName: ApproximateNumberOfMessagesVisible
      Namespace: AWS/SQS
      Statistic: Average
      Period: 300
      EvaluationPeriods: 2
      Threshold: 1000
      ComparisonOperator: GreaterThanThreshold
      Dimensions:
        - Name: QueueName
          Value: !GetAtt ProcessingQueue.QueueName

  DLQDepthAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: sqs-dlq-not-empty
      MetricName: ApproximateNumberOfMessagesVisible
      Namespace: AWS/SQS
      Statistic: Average
      Period: 300
      EvaluationPeriods: 1
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      Dimensions:
        - Name: QueueName
          Value: !GetAtt DeadLetterQueue.QueueName

  ProcessingTimeAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: sqs-message-age-high
      MetricName: ApproximateAgeOfOldestMessage
      Namespace: AWS/SQS
      Statistic: Maximum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 600  # 10 minutes
      ComparisonOperator: GreaterThanThreshold
      Dimensions:
        - Name: QueueName
          Value: !GetAtt ProcessingQueue.QueueName

Outputs:
  QueueUrl:
    Value: !Ref ProcessingQueue
    Description: Queue URL
  QueueArn:
    Value: !GetAtt ProcessingQueue.Arn
    Description: Queue ARN
  DLQUrl:
    Value: !Ref DeadLetterQueue
    Description: Dead-Letter Queue URL

Key Elements:

• Standard queue for high throughput
• Dead-letter queue for failed messages
• Long polling (10s) to reduce costs
• Visibility timeout matches processing time
• Message deduplication via timestamp
• KMS encryption
• Lambda with batch processing
• Alarms for queue depth and age

Pattern 2: FIFO Queue for Ordered Processing

Resources:
  # FIFO Queue (message order guaranteed)
  OrderQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: order-processing.fifo
      FifoQueue: true
      ContentBasedDeduplication: false  # Explicit IDs preferred
      VisibilityTimeout: 300
      MessageRetentionPeriod: 345600
      ReceiveMessageWaitTimeSeconds: 10
      DeduplicationScope: messageGroup  # Dedup per group
      FifoThroughputLimit: perMessageGroupId

  # FIFO Dead-Letter Queue
  OrderDLQ:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: order-dlq.fifo
      FifoQueue: true
      ContentBasedDeduplication: true
      MessageRetentionPeriod: 1209600

  # Link DLQ
  OrderQueueWithDLQ:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: order-queue.fifo
      FifoQueue: true
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt OrderDLQ.Arn
        maxReceiveCount: 3

Key Elements:

• FIFO queue for strict ordering
• MessageGroupId for partitioning (same group = same order)
• ContentBasedDeduplication for automatic dedup
• Deduplication scope per group
• Throughput limit per message group

Pattern 3: Batch Processing with Visibility Adjustment

Resources:
  # Batch Processing Lambda
  BatchProcessorFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: batch-processor
      Runtime: python3.11
      Handler: index.handler
      Timeout: 600  # 10 minutes for batch
      Code:
        ZipFile: |
          import boto3
          import json
          from datetime import datetime, timedelta
          
          sqs = boto3.client('sqs')
          
          def handler(event, context):
              queue_url = event['QueueUrl']
              
              # Receive batch of messages
              response = sqs.receive_message(
                  QueueUrl=queue_url,
                  MaxNumberOfMessages=10,
                  WaitTimeSeconds=10
              )
              
              messages = response.get('Messages', [])
              
              try:
                  # Process batch
                  process_batch(messages)
                  
                  # Delete processed messages
                  for msg in messages:
                      sqs.delete_message(
                          QueueUrl=queue_url,
                          ReceiptHandle=msg['ReceiptHandle']
                      )
                  
              except Exception as e:
                  # Extend visibility for slower processing
                  for msg in messages:
                      sqs.change_message_visibility(
                          QueueUrl=queue_url,
                          ReceiptHandle=msg['ReceiptHandle'],
                          VisibilityTimeout=600  # +10 minutes
                      )
                  raise
              
              return {'processed': len(messages)}
          
          def process_batch(messages):
              # Batch processing logic
              pass

Key Elements:

• Batch receive for efficiency
• Dynamic visibility adjustment
• Graceful failure handling
• Message deletion on success

Integration Approaches

1. Integration with Lambda

Lambda + SQS enables:

• Event-driven processing
• Batch message handling
• Automatic scaling

2. Integration with SNS

SNS + SQS enables:

• Fan-out pattern
• Persistence layer
• Multiple consumers

3. Integration with Application Load Balancer

ALB + SQS enables:

• HTTP request queueing
• Backend decoupling
• Request buffering

Common Pitfalls

❌ Pitfall 1: Visibility Timeout Too Short

Problem: Messages re-delivered before processing completes; duplicates.

Solution:

• Set to match longest processing time
• Use change_message_visibility if processing slower

❌ Pitfall 2: No Dead-Letter Queue

Problem: Poison pill messages cause infinite retries.

Solution:

• Configure DLQ with maxReceiveCount
• Monitor DLQ for failures

❌ Pitfall 3: Long Polling Disabled

Problem: Empty responses waste API calls; high costs.

Solution:

• Enable long polling (WaitTimeSeconds = 10s)
• Reduces API call costs by 80%+

❌ Pitfall 4: Not Deleting Processed Messages

Problem: Messages re-delivered indefinitely.

Solution:

• Always delete on successful processing
• Only delete after message handle no longer valid

❌ Pitfall 5: FIFO for Non-Ordered Workloads

Problem: FIFO throughput limited (300 msg/s per group).

Solution:

• Use Standard queue for high throughput
• FIFO only when order matters

Best Practices Summary

Related Skills

Core Workflow

1. Assess Requirements — Understand the use case, scale, integration needs, and existing infrastructure. Checkpoint: Document requirements, constraints, and success criteria.

2. Design Architecture — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. Checkpoint: Verify the architecture addresses all requirements and follows CNCF conventions.

3. Implement & Configure — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. Checkpoint: Validate all YAML against schema and test in a staging environment.

4. Deploy & Monitor — Apply manifests to the cluster, verify component health, and confirm observability is working. Checkpoint: Confirm all pods/services are running, probes passing, and metrics/alerts configured.

Constraints

MUST DO

• Include at least one complete working YAML manifest example
• Note when content is auto-generated vs. manually verified
• Reference relevant CNCF project documentation

MUST NOT DO

• Deploy manifests without testing in a staging environment first
• Use deprecated API versions (e.g., apps/v1beta1)
• Omit resource limits and requests in Kubernetes manifests

Live References

Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content.

• Primary Documentation
• API Reference or Getting Started
• Configuration Guide
• Best Practices
• Common Patterns or Tutorials