Amazon Kinesis

In today’s data-driven applications, we often need to process continuous data streams, like sensor data from IoT devices, website clickstreams, or logs from distributed systems. Amazon Kinesis is a suite of services that Amazon Web Services (AWS) offers for collecting, processing, and analyzing real-time streaming data. It’s designed to handle high volumes of data from various sources, such as website clicks, social media feeds, sensor data, financial transactions, etc.

Amazon Kinesis services

Amazon Kinesis is serverless and allows us to define how we want to process the data streams and integrate the processed data with other AWS services for further analysis. Here are four primary services under the Amazon Kinesis that we’ll cover in this lesson:

• Kinesis Data Streams allows us to build real-time applications that consume high-throughput data with millisecond latency. We write data to a stream using AWS SDKs or the AWS CLI, then process that data using consumers like Lambda functions, EC2 applications, or custom code. The illustration below shows a stream produced by web servers called stream producers, which is ingested by Kinesis Data Streams to process them and send them to EC2 or Lambda, the stream consumers in this case:

• Kinesis Data Firehose simplifies the delivery of real-time streams to destinations like Amazon S3, Redshift, OpenSearch, or even custom HTTP endpoints. It handles buffering, batching, retrying, and optional transformation using AWS Lambda. The illustration below shows Kinesis Data Firehose using processed data packages from Kinesis Data Streams and delivering them to the warehouse destination service, like S3. Consumer applications in EC2 instances then retrieve packages from the warehouses for further processing or analysis.

• Kinesis Video Streams is a specialized service for ingesting and processing video data from connected devices such as ...