Messaging Service:
Direct communication (Synchronous)
Communication through queue in between service (Asynchronous)
Synchronous can be problematic in case of sudden spike in traffic.
Ex: a video uploading service which accepts 10 videos at one time & all of sudden 1000 videos are uploaded which will lead to service crash.
So in this case decoupling cas solve this problem
Using SQS → queue model
Using SNS → pub/sub model
Kinesis → real time streaming model
These services can scale independently from our application.
SQS: Simple Queuing Service
Here producer will send/push the messages to SQS Queue and Consumer will poll the messages from the SQS Queue.
SQS Standard Queue:
One of the oldest Services in AWS (10 years old).
Fully managed service, to decouple the application.
Can have duplicate messages (at least once delivery, occasionally).
Can have out of order messages (best effort ordering).
Attributes:
Unlimited throughput, unlimited number of messages in queue.
Default retention of messages 4 days and upto 14days.
Low latency (<10ms on publish and receive).
Limitation of 256Kb per message sent.
SQS Producer Messages:
Producer to SQS using SDK (sendMessage API).
The message is persisted in the queue until the consumer reads it & deletes it.
Default retention of messages 4 days and upto 14days.
Example: e-commerce website (send an order to be processed)
SQS Consumers Messages:
Consumer (running ec2 instance or lambda function or even on prem instance).
Consumer polls for messages to SQS Queue.
Consumers can receive up to 10 messages at a time.
Process the message (ex: insert the message into RDS).
Delete the messages from the queue using DeleteMessage API
SQS Message Visibility Timeout:
When one consumer consumes this message it will be unavailable for another consumer till message visibility timeout.
The first consumer can also extend the visibility timeout by calling ChangeMessageVisibility API.
If the visibility timeout is high(hour) and if a consumer crashes it can reprocess the message.
If the visibility timeout is low(seconds) then the chances of duplicate message are high
SQS Dead letter Queue:
If the message fails to process within message visibility timeout then the message is sent back to the queue.
We can set the threshold on how many times this message can be sent back to the queue.
After the MaximumReceives threshold is exceeded the message goes into the dead letter queue.
The Dead letter queue is useful for debugging purpose
Make sure to process the dead letter queue before it expires.
We can set the retention period to this dead letter queue.
SQS Delay Queue:
Delay a message up to 15mins (Consumers don't see it immediately).
Default is set to 0 (Delivery Delay)
Can set default to queue level
Can override the default value on send using DelaySeconds parameter.
SQS FIFO
First In First Out (Ordering of message in a queue).
Ordering guarantee.
Exactly once send capacity (by removing duplicates).
Messages are processed in order by a consumer.
Limited throughput: 300 msg/s & without batching 3000 msg/s
SQS with Auto Scaling Group:
SNS (Simple Notification Service):
Pub/Sub Service: meaning publish and subscriber service.
The event producer only sends message to one SNS topic.
As many event receivers(subscribers) we want to listen to SNS topic notifications.
Each subscriber to the topic will receive all the messages. (new feature to filter out the messages but by default every one will receive it)
Upto 10million subscribers per topic (high scale)
1lac topic limit.
Subscribers can be (Protocol):
SQS
HTTP/HTTPS
Lambda functions
Email notifications
SMS messages
Mobile notifications
SNS Integrates with lot of AWS services:
SQS
Cloud Watch for alarms
Amazon s3 (on bucket events)
Auto Scaling group event.
CloudFormation (upon stack changes) etc…
How to publish SNS message:
Topic Publish:(using SDK).
Topic Creation.
Create a Subscription
Publish to the topic
Direct Publish: (For mobile apps SDK)
Create a platform application
Create a platform endpoint
Publish to the endpoint
Works with Google GSN, Apple APNS, Amazon ADM etc.
SNS Security:
Inflight Encryption
At rest encryption using KMS
Client side encryption
Access control using IAM policies to regulate access to the SNS API.
You can define SNS policies similar to S3 bucket policies
Useful for cross account access to SNS topic
Useful for other services to write to SNS topics (S3).
SQS + SNS Fan Out:
Push once into SNS and receive all in SQS queues that are SNS subscribers
Fully decoupled no data loss
SQS Allows for: data persistence, delayed processing and retries of work.
To make this thing work the SQS needs to have allow policy to give access to SNS to write
* Important Note: SNS cannot send messages to SQS FIFO Queues (AWS Limitation)
If asked in exam about this simply rule out since it’s not possible as of today:
AWS Kinesis:
Kinesis is managed alternative to Apache Kafka
Real time Big data collection tool.
Great for streaming processing frameworks like Spark, NiFi etc.
Data is automatically replicated to 3 AZ
Three sub products:
Kinesis Streams: low latency streaming ingest at scale.
Kinesis Analytics: This is to perform real time analytics on streams using SQL.
Kinesis Firehose: Load streams into s3, RedShift, ElasticSearch etc.
On a high level kinesis is in the middle to stream and then to perform analytics over it and then to store somewhere for long time
#TODO need to add remaining features of Kinesis
.
No comments:
Post a Comment