Search⌘ K
AI Features
Home
Courses
Grokking the AWS Certified Machine Learning Engineer - Associate

Streaming Data Architectures

Understand how streaming data pipelines support real-time machine learning on AWS. Learn to use Kinesis Data Streams, Data Firehose, Managed Streaming for Kafka, and Apache Flink to ingest, transform, and deliver data for scalable ML workflows. This lesson guides you through designing, monitoring, and optimizing streaming architectures for real-time inference and batch training scenarios.

We'll cover the following...

Streaming data pipelines form the backbone of real-time ML applications on AWS. Whether you are building a fraud detection system that must score transactions in milliseconds or a recommendation engine that adapts to user behavior as it happens, batch processing alone cannot meet latency demands. For the AWS Certified Machine Learning Engineer – Associate exam, you need to understand how four core services work together to move high-velocity data from producers to ML models with minimal delay and maximum reliability.

These four services divide the streaming problem into distinct responsibilities:

  • Amazon Kinesis Data Streams captures gigabytes of continuous data per second for real-time consumption.

  • Amazon Data Firehose (formerly Amazon Kinesis Data Firehose) automates delivery into data lakes and warehouses with built-in format conversion.

  • Amazon Managed Streaming for Apache Kafka (MSK) provides fully managed Kafka clusters for teams already invested in the open-source Kafka ecosystem.

  • Managed Service for Apache Flink sits between ingestion and destinations, performing windowed aggregations, feature engineering, and anomaly scoring on data in motion.

In most streaming architectures, the final destination is Amazon S3, where data lands in optimized columnar formats such as Apache Parquet. From S3, downstream services such as SageMaker training jobs and Athena queries consume the data efficiently, closing the loop between real-time ingestion and model development.

The following diagram illustrates how these services connect in a typical end-to-end streaming ML pipeline.

AWS streaming data architecture with parallel ingestion paths converging on S3 data lake for ML training and real-time inference
AWS streaming data architecture with parallel ingestion paths converging on S3 data lake for ML training and real-time inference

The AWS architecture above illustrates an end-to-end streaming data pipeline that captures real-time events (such as IoT telemetry, clickstreams, and application logs) using Amazon Kinesis Data Streams (KDS) and Amazon MSK. The ingested data is processed through two distinct paths: Apache Flink handles complex, real-time stream processing, while Amazon Data Firehose manages micro-batching and format conversion before delivering the data to an Amazon S3 data lake. Finally, AWS Glue catalogs the underlying schemas, enabling Amazon SageMaker to consume both historical training data from S3 and real-time features from Flink to power machine learning workflows.

With this architectural overview in place, let’s examine each service in detail, starting with the real-time ingestion layer.

Amazon Kinesis Data Streams

Kinesis Data Streams (KDS) is a real-time data-streaming service that captures gigabytes of data per second from hundreds of thousands of ...