Messaging with SQS and SNS
Learn how to use Amazon SQS and SNS to build scalable, decoupled, and fault-tolerant messaging solutions for applications.
In this lesson, we'll explore Amazon Simple Queue Service (Amazon SQS) and Amazon Simple Notification Service (Amazon SNS) as tools for implementing scalable and decoupled application messaging. We examine their core features, messaging patterns, and how we can integrate these services into serverless architectures.
Introduction to messaging in AWS
Modern application architectures use asynchronous messaging to improve scalability, fault isolation, and decoupling. In AWS, Amazon SQS and Amazon SNS provide essential messaging functionality.
Amazon SQS is a pull-based queueing service that stores messages until consumers/receivers are ready to process them. It helps decouple microservices, batch processing systems, and other distributed components.
Amazon SNS is a push-based publish/subscribe (pub/sub) service that delivers messages to multiple subscribers. It supports multiple protocols such as HTTPS, Lambda, email, SMS, and Amazon SQS.
Asynchronous messaging via Amazon SQS and Amazon SNS contrasts with synchronous invocation patterns, such as using Amazon API Gateway to trigger AWS Lambda. These patterns are used when durability, buffering, and loose coupling are needed. We can choose synchronous patterns when we need a response before continuing.
Amazon SQS: Message queues
Amazon Simple Queue Service (SQS) is a message queuing service that can help us decouple our application components. It uses queues to store the messages sent by producers and sends them to consumers using a polling model. Consumers check the queues for messages sent by the producer whenever they are ready to process them.
Amazon SQS uses redundant infrastructure and stores the messages it receives in multiple availability zones, making SQS queues highly available and durable. Unprocessed messages can be retained for a maximum of 14 days, after which all copies of the message are deleted from the SQS servers.
In SQS, it is the consumer's responsibility to delete the messages available in an SQS queue after processing them. Once a message “M1” is received by a consumer, SQS starts a visibility timeout period during which this message is not sent to any other consumer to avoid message reprocessing.
Types of SQS queues
In many scenarios, different applications have varying messaging needs; some prioritize strict ordering, while others focus on high throughput and at-least-once delivery. To address these needs, Amazon SQS offers multiple queue types tailored for different use cases. It supports two queue types:
1. Standard queues
In standard queues, messages are delivered at least once to each consumer. In this type of queue, the messages are stored in multiple availability zones, so there is a chance that multiple copies of one message are delivered. In some rare cases, we may delete a message whose copy is stored on an unavailable server. The message will become visible for processing once again when the server is available. Due to this, the order of delivery of the messages is not guaranteed in standard queues, and it is recommended to use them in applications where processing one message multiple times doesn’t affect it, such as sending email notifications or updating analytics counters.
Here's how we can create a standard queue using AWS CLI and AWS SDK:
aws sqs create-queue \--queue-name MyStandardQueue \--attributes VisibilityTimeout=30,MessageRetentionPeriod=1209600
The VisibilityTimeout=30 argument sets how long a message remains invisible after being received and the MessageRetentionPeriod=1209600 argument retains unprocessed messages for 14 days.
Once we’ve created a standard queue, producers can begin sending messages to it. Standard queues offer at-least-once delivery and best-effort ordering. The following example shows how to send a message to a standard queue using AWS CLI and Boto3.
aws sqs send-message \--queue-url https://sqs.<Reegion>.amazonaws.com/<Account-ID>/MyStandardQueue \--message-body "Hello from standard queue"
2. FIFO queues
In FIFO queues, messages are delivered to consumers exactly once, and their order of delivery is maintained. Due to this ...