Part 7. Learn about AWS at once - SQS, SNS, Kinesis

Asynchronous delivery can help avoid traffic issues that come with synchronous communication. So, in AWS, you can use SQS, SNS, and Kinesis as a messaging method to deliver messages asynchronously and according to the occurrence of events.

SQS (Simple Queue Service)

Standard Queue

When the producer delivers a message through the queue method (send), the message is received (poll) from the consumer. By default, it has a retention period of 4 days and can be stored for up to 14 days. Duplicate messages may occasionally occur and messages are not guaranteed to be in order.

• There is a maximum size limit of 256KB per message.
• If you want to send a message larger than the limited amount, send it using SQS Extended Client.
• You can check with CloudWatch whether there are many messages in the queue and adjust the number of EC2 instances with ASG.
• You can also separate applications by placing SQS between the frontend and backend.
• Secure it by encrypting it or adjusting IAM and SQS access permissions. -API
• CreateQueue, PurgeQueue, SendMessage, ReceiveMessage, DeleteMessage, MaxNumberOfMessage(max 10), ReceiveMessageWaitTimeSeconds, ChangeMessageVisibility

message visibility timeout

The message visibility timeout is 30min by default, and at this time, this message is not visible to other consumers. You will be able to view the message after 30 minutes. Consumers can receive additional time by calling the ChangeMessageVisibility API.

• If the visibility timeout is too long, reprocessing may be delayed.
• If the visibility timeout is too short, you may receive duplicate messages.

Dead Letter Queue (DLQ)

If the poll fails due to message visibility timeout, several retries are made. And you can limit the number of schedule retries. We recommend setting the retention period for DLQ to 14 days.

• After modifying the code, the work in DLQ can be redriven back to SQS.

Delay Queue

Rather than polling messages right away, you can delay them for up to 15 minutes. The default value is set to 0sec so that it can be polled immediately.

Long Polling

From the consumer's perspective, they can wait if they do not receive a message. In this case, the recommended waiting time is 20 seconds (1 to 20 seconds). Can be enabled with ReceiveMessageWaitTimeSeconds.

FIFO Queue

This is a queue in which the first message that comes in is polled first. Unlike standard queues, there is a throughput limit and order.

Deduplication

Deduplication is possible, with an interval of 5 minutes. Removal methods include using a SHA-256 hash based on content and explicitly providing a message deduplication ID.

Message Grouping

If they have the same MessageGroupID, FIFO Queue has only one consumer for that group, and each consumer has a different consumer.

SNS (Simple Notification Service)

SNS is a service that publishes messages in a Pub/Sub manner and delivers them to multiple subscribers. In other words, there is one message and multiple receivers.

• Up to 12,500,000 subscriptions can be made to one topic.
• The topic limit is 100,000.
• The maximum message size sent at one time is 256KB.
• Issuers may include various AWS, CloudWatch, AWS Budget, Lambds, S3, ASG, DynamoDB, etc.
• Subscribers can be SQS, Lambda, Kinesis, HTTP endpoint, SMS, Email, etc.
• Security methods apply the same methods as SQS.
• Can be used with SQS. When SQS receives an event, it delivers messages from each queue to multiple services.

FIFO Topic

Here too, there is a FIFO Topic method that can deliver messages in a first-in, first-out manner. Messages are sent in order and throughput is limited the same as SQS.

Filtering Policy

When messages are sent to SNS Topic subscribers, they can be filtered. Without filters, subscribers will receive all messages.

Kinesis

Kinesis is a large-scale real-time data streaming platform used to collect and process data in real time. It is useful for collecting, processing, or analyzing data in real time.

• Kinesis Data Streams
• Kinesis Data Firehose
• Kinesis Data Analytics
• Amazon Kinesis Video Streams

Kinesis Data Streams

Kinesis Data Streams is a service designed for real-time data streaming, allowing data to be ingested into streams and processed simultaneously by multiple consumers. It is mainly used for log data, real-time analysis, and monitoring.

• Retention can be set between 1 and 365 days.
• Once data enters Kinesis, it is permanent and cannot be deleted.

Capacity Modes

• Provisioned mode
• You can choose the number of shards and adjust their scale.
• Pay per hour for the number of shards provisioned.
• A shard processes 1MB of input per second or 1000 records per second.
• On-demand mode
• It does not require provision and has basic capacity, so there is no need for separate management.
• Automatically scales based on observed throughput peaks over the last 30 days.
• Pay per hour for streams and per GB for data input and output.

ProvisionedThroughputExceeded

This problem occurs when the provisioned throughput set by the user is exceeded. Solutions include increasing the number of shards or distributing partition keys evenly. There is also a way to perform a backoff that continuously increases the retry period.

Consumer Type

Shared (Classic) Fan-out Consumer (Pull method)

It is primarily used in schemes where the number of consumers is small, when multiple consumers are pulling data from the same stream, each consumer is limited by the rate of data available from a single shard. Therefore, although its performance may have certain limitations, it is relatively inexpensive.

Enhanced Fan-out Consumer (Push method)

Kinesis is expensive to transfer data in real time. Data is pushed to HTTP/2.

Kinesis Client Library (KCL)

A library that helps multiple application instances read data from a Kinesis stream in parallel. Each shard can only be read by one KCL instance, which prevents duplicate reads and distributes work efficiently. The progress of records is stored as checkpoints in DynamoDB, and KCL can run in multiple environments (EC2, Elastic Beanstalk, on-premises). KCL 1.x supports only shared consumer, and KCL 2.x supports enhanced fan-out to further improve performance.

Operation

Shard Splitting

Use this when you want to increase stream capacity by dividing shards. A commonly used shard called a "hot shard" is divided into two or more shards.

Merging Shards

In contrast to the above, shards are merged, and “cold shards” with low traffic are merged. You cannot combine two or more shards into one.

Kinesis Data Firehose

Kinesis Data Firehose is a service that collects data in near real-time and delivers it directly to AWS services (e.g. S3, Redshift, Elasticsearch, etc.). It can automatically load and process data from a stream, making it useful for storing and analyzing data. Unlike Kinesis Data Streams, Firehose delivers stream data directly to AWS services, is fully managed, and provides automatic scaling. Because it moves directly to the destination, a separate retry function is not provided.

• Real-time analysis is possible using Kinesis Data Analytics for Apache Flink.
• When sending data to Kinesis, if you send data using a partition key, the same key will always go into the same shard.
• Compared to SQS FIFO, Kinesis can have as many consumers in parallel as the number of shards, and SQS FIFO can have as many consumers as the number of group IDs.