DVA-C02 Preparation Material 5 - Event Handling & Data Streaming

###EventBridge

• You can set the main account's permissions to have all accounts send events to the main account's EventBridge.
• Direct delivery from each account

###SQS

• Maximum message size 256KB
• Useful for storing and using up to 2GB messages using SQS Extended Client
• Consider S3 unless you store data in a queue.
• The queue's visibility timeout value is in seconds, the default is 30 seconds (an integer from 0 to 43,200 (12 hours)).
• There is no maximum storage message limit
• The limit on in-flight messages (messages that have been received from the queue by the consumer but not deleted from the queue) is set at a maximum of approximately 120,000.
• Queue type cannot be changed after creation
• Queue tags are case sensitive
• Dead letter queues in FIFO queues must also be FIFO queues. Likewise, a standard queue's Dead-Letter Queue must also be a standard queue.
• Message isolation for handling unhandled errors
• Handling of Lambda that fails asynchronous invocation and all retries
• DeleteQueue - Delete the queue and all its contents after testing SQS.
• PurgeQueue - Deletes messages from the specified queue and leaves the queue intact.
• DelaySeconds - The number of seconds to delay delivery of all messages in the queue.
• Delay Queue created
• MessageRetentionPeriod - Controls the number of seconds that Amazon SQS retains messages.
• Activate SQS KMS
• You can choose to have SQS encrypt messages stored in standard and FIFO queues using encryption keys provided by AWS Key Management Service (KMS).
• ApproximateNumberOfMessagesVisible
• Amazon SQS-based scaling policy
• Number of messages waiting in the SQS queue without being processed
• Comparison of per-instance backlog and per-instance tolerable backlog
• Backlog per instance
• ApproximateNumberOfMessagesVisible / number of instances
• Acceptable backlog per instance
• It is necessary to consider both the number of messages, processing time and acceptable delay time.
• Acceptable delay time/message processing time
• Backlog per instance > Allowed backlog
• Add instances to speed up message processing.
• Backlog per instance
• Reduce instances to avoid excessive resources.
• Target Tracking Scaling Policy
ASG processing when
• Target Value is reached
• Polling
• Default has short polling
• Polling refers to the process of periodically checking whether new messages have arrived in the message queue
• The cost of using SQS can be minimized by using long polling
• Highly scalable, no need to take any action to handle large amounts of work
• Finding the rolling average
• Set Lambda's reserved concurrency to 1 (1 instance running)
• Calculate rolling average in Lambda
• Store the calculated rolling average in Amazon ElastiCache

SNS

• Failure handling after SNS notification can be handled with SQS
• Receive SNS order creation notifications from SQS queue
• SNS topic filters allow notifications to be sent to necessary subscribers
• SNS + Lambda
• In this case, you cannot have Lambda handle EC2 instance order fulfillment >> Serverless limitations

S.E.S.

• Use Exponential Backoff to resolve Throttling – Maximum sending rate exceeded error
• Instead of aggressively retrying immediately, the client waits a period of time between attempts.
• Requests rejected with a Throttling error can be retried later and are likely to succeed.

Kinesis

Kinesis Data Stream

• Provides greater flexibility for integrating downstream applications than Kinesis Data Firehose and is relatively inexpensive
• Kinesis Data Stream focuses more on collection and storage, while Kinesis Data Firehose focuses on processing and transmission.
• PutRecords API
• Create multiple records in a single call
• Up to 500
• ProvisionedThroughputExceededException
• The request rate for the stream is too high or the requested data is too large for the available throughput.
• "Hot" partition
• Reduce request frequency or size
• Use error retry and exponential backoff mechanisms
• Provide sufficient capacity by increasing the number of shards within the data stream

Kinesis Data Firehorse

• Service for real-time streaming data transmission to destinations such as S3, Elasticsearch Service, Redshift, and Splunk
• ElastiCache is not supported
• Kinesis Data Firehose supports using AWS Lambda functions to transform data before delivering it to the destination.

Kinesis Data Analytics

• Used to build SQL queries and sophisticated Java applications

  Kinesis Services	Description	Key Features	Use
  Kinesis Data Streams	Services that capture and process real-time streaming data	- Data is captured as a stream and processed by multiple consumers - Auto-scalable - Continuous data processing	- Real-time analysis - Log data streaming processing - Event-based application
  Kinesis Data Firehose	Stream data in real time to S3, Redshift, Elasticsearch, and more	- Automatically transmits and loads data - Provides data conversion function - Transmits data to multiple targets	- Real-time data loading and analysis - ​​Logging and monitoring data transmission
  Kinesis Data Analytics	Service that analyzes Kinesis streams and Firehose data in real time	- Real-time data processing with SQL queries - Real-time analysis and processing of stream data - Can be linked with Lambda	- Real-time data streaming analysis - Building a dashboard update and notification system
  Kinesis Video Streams	Services that capture, store, and process video streaming data	- Real-time video streaming and data capture - Integrates with Amazon Rekognition - Secure video data storage	- Video analysis and monitoring - Data collection from IoT cameras

---

ETC

• Deploy scalable Oracle Real Application Clusters (RAC) on Amazon EC2 using Amazon Machine Images (AMI) from AWS Marketplace
• Cost budget alerts require 5 weeks of usage data for forecasting
• CMS (Content Management System) application is a software application that creates, manages, edits, and stores digital content.
• WordPress, Wix. etc.
• Use AWS CLI --dry-run option
• The --dry-run option checks whether the operation has the necessary permissions without actually requesting it and provides an error response.
• If you have the required permissions, the error response is DryRunOperation
• Otherwise UnauthorizedOperation
• Create a public SSH key from your personal SSH key and then use it in a new region.