SNS / SQS
SNS is a distributed publish-subscribe system. Messages are pushed to subscribers as and when they are sent by publishers to SNS.
SQS is distributed queueing system. Messages are not pushed to receivers. Receivers have to poll or pull messages from SQS. Messages can't be received by multiple receivers at the same time. Any one receiver can receive a message, process and delete it. Other receivers do not receive the same message later. Polling inherently introduces some latency in message delivery in SQS unlike SNS where messages are immediately pushed to subscribers. SNS supports several end points such as email, SMS, HTTP end point and SQS. If you want unknown number and type of subscribers to receive messages, you need SNS.
You don't have to couple SNS and SQS always. You can have SNS send messages to email, SMS or HTTP end point apart from SQS. There are advantages to coupling SNS with SQS. You may not want an external service to make connections to your hosts (a firewall may block all incoming connections to your host from outside).
Your end point may just die because of heavy volume of messages. Email and SMS maybe not your choice of processing messages quickly. By coupling SNS with SQS, you can receive messages at your pace. It allows clients to be offline, tolerant to network and host failures. You also achieve guaranteed delivery. If you configure SNS to send messages to an HTTP end point or email or SMS, several failures to send message may result in messages being dropped.
SQS is mainly used to decouple applications or integrate applications. Messages can be stored in SQS for a short duration of time (maximum 14 days). SNS distributes several copies of messages to several subscribers. For example, let’s say you want to replicate data generated by an application to several storage systems. You could use SNS and send this data to multiple subscribers, each replicating the messages it receives to different storage systems (S3, hard disk on your host, database, etc.).
| SQS | SNS | |
|---|---|---|
| Entity Type | Queue (Similar to JMS) | Topic (Pub/Sub system) |
| Message consumption | Pull Mechanism - Consumers poll and pull messages from SQS. Messages can’t be received by multiple receivers at the same time. Any one receiver can receive a message, process and delete it. Other receivers do not receive the same message later. | Push Mechanism - SNS Pushes messages to consumers |
| Use Case | Decoupling two applications and allowing parallel asynchronous processing | Fanout - Processing the same message in multiple ways |
| Persistence | Messages are persisted for some (configurable) duration if no consumer is available (maximum two weeks), so the consumer does not have to be up when messages are added to queue. | No persistence. Whichever consumer is present at the time of message arrival gets the message and the message is deleted. If no consumers are available then the message is lost after a few retries. |
| Consumer Type | All the consumers are typically identical and hence process the messages in the exact same way (each message is processed once by one consumer, though in rare cases messages may be resent) | The consumers might process the messages in different ways |
| Sample applications | Jobs framework: The Jobs are submitted to SQS and the consumers at the other end can process the jobs asynchronously. If the job frequency increases, the number of consumers can simply be increased to achieve better throughput. | Image processing. If someone uploads an image to S3 then watermark that image, create a thumbnail and also send a "Thank You" email. In that case S3 can publish notifications to an SNS topic with three consumers listening to it. The first one watermarks the image, the second one creates a thumbnail and the third one sends a "Thank You" email. All of them receive the same message (image URL) and do their processing in parallel. |
Copied from this Stack Overflow post
Fan out architecture
Fanout is a messaging design where the published message from a particular publisher is consumed by multiple different subscribers independently and simultaneously. The intention is that the same published message will be consumed by different consumers and be processed in different ways.
The use case of fanout pattern is applicable in places where a publisher needs to asynchronously communicate to multiple consumers on a single workload.
An example could be: Let’s say in an online library system, a microservice is responsible for placing the borrowing request for a book – Book Rent service. Here, once a book is successfully rented out to a customer, this service can act as a publisher and produce a message for other services that are responsible for (i) Removing this from the customer’s ‘Wishlist’, (ii) Reducing the availability count for this book in the online library shelf, (iii) Updating the “Currently Reading Shelf” of the customer. These consumer services would listen for the same message in different queues and can work on this notification independently without making the renting process to wait. Here the publisher, “fans out” the message in different queues where interested consumers are hooked into it for consuming the message. The publisher here, the Book Rent service, doesn't need to wait until other services are finished acting on the message and can seamlessly and independently issue the book to the customer and carry out more renting requests making it available for the users.
For our app, we will need fanout because without it, we can't implement services that need to do work in parallel, we would have to go the old way of doing "polling" and scanning database for changes. Instead, we need to be event driven.
Especially since we are using Lambdas. When using Lambda in a serverless architecture, the goal should be to design tightly focused functions that do one thing and do it well. When these functions are composed to accomplish larger goals in microservice architectures, the complexity shifts from the internal components to the external communication between components. It’s all too easy to accidentally back into an architecture that is rigid to change because components are too knowledgeable of each other via the communication paths between them.
If in the future, a broker creates a Load Posting (for booking purposes), we may have multiple services that need to act on that event. By using fanout, we can have multiple subscribers on it. For example, a queue can be there that subscribes to a "new posting" and notifies carriers that are intersted in the load. Another queue is responsible for adding the posting to all the carriers that follow that broker.
Serverless Example
See here for full article.
# Create the subscription to the 'secondQueue'
secondQueueSubscription:
Type: 'AWS::SNS::Subscription'
Properties:
TopicArn: !Ref mySnsTopic
Endpoint: !GetAtt
- secondQueue
- Arn
Protocol: sqs
FilterPolicy:
action:
- 'sendSMS'
RawMessageDelivery: 'true'
# Resources section of your serverless.yml file
resources:
Resources:
### PART ONE: Create SNS Topic and SQS Queues
# Create our SNS Topic
mySnsTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: ${self:service}-${self:provider.stage}-my-sns-topic
# Create our 'firstQueue' SQS queue
firstQueue:
Type: 'AWS::SQS::Queue'
Properties:
QueueName: ${self:service}-${self:provider.stage}-first-queue
RedrivePolicy:
deadLetterTargetArn: !GetAtt
- firstQueueDLQ
- Arn
maxReceiveCount: 3
# Create our 'firstQueue' Dead Letter Queue SQS queue
firstQueueDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:service}-${self:provider.stage}-first-queue-dlq
# Create our 'secondQueue' SQS queue
secondQueue:
Type: 'AWS::SQS::Queue'
Properties:
QueueName: ${self:service}-${self:provider.stage}-second-queue
RedrivePolicy:
deadLetterTargetArn: !GetAtt
- secondQueueDLQ
- Arn
maxReceiveCount: 3
# Create our 'secondQueue' Dead Letter Queue SQS queue
secondQueueDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:service}-${self:provider.stage}-second-queue-dlq
### PART TWO: Create SQS Queue Policies
# Create our queue policy for the 'firstQueue'
snsToFirstQueueSqsPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: 'allow-sns-messages'
Effect: Allow
Principal: '*'
Resource: !GetAtt
- firstQueue
- Arn
Action: 'SQS:SendMessage'
Condition:
ArnEquals:
'aws:SourceArn': !Ref mySnsTopic
Queues:
- Ref: firstQueue
# Create our queue policy for the 'secondQueue'
snsToSecondQueueSqsPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: 'allow-sns-messages'
Effect: Allow
Principal: '*'
Resource: !GetAtt
- secondQueue
- Arn
Action: 'SQS:SendMessage'
Condition:
ArnEquals:
'aws:SourceArn': !Ref mySnsTopic
Queues:
- Ref: secondQueue
### PART THREE: Subscribe our SQS Queues to our SNS Topic
# Create the subscription to the 'firstQueue'
firstQueueSubscription:
Type: 'AWS::SNS::Subscription'
Properties:
TopicArn: !Ref mySnsTopic
Endpoint: !GetAtt
- firstQueue
- Arn
Protocol: sqs
RawMessageDelivery: 'true'
# Create the subscription to the 'secondQueue'
secondQueueSubscription:
Type: 'AWS::SNS::Subscription'
Properties:
TopicArn: !Ref mySnsTopic
Endpoint: !GetAtt
- secondQueue
- Arn
Protocol: sqs
RawMessageDelivery: 'true'