CLOUD LABS
Getting Started with Amazon Managed Streaming for Apache Kafka
In this Cloud Lab, you’ll create an Amazon MSK cluster and a client machine using EC2, giving it access to our MSK cluster through an IAM role. You’ll then create a topic in our cluster and add producers and consumers to it.
beginner
Certificate of Completion
Learning Objectives
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is an Amazon managed service that allows you to run applications that use Apache Kafka as a communication system. Through this service, you can configure your clusters and launch brokers in various availability zones. In case a server or broker fails, Amazon MSK provides automatic failure detection and recovery. CloudWatch logs and alarms can also be created to monitor the clusters and brokers created through MSK.
In this Cloud Lab, you’ll first create a VPC and a security group. You’ll then create an MSK cluster and configure it to launch one broker per availability zone. After this, you’ll attach an IAM role to an EC2 instance to give it permission to access the cluster you created. Finally, you’ll use the EC2 instance to launch a Kafka topic and add producers and consumers to this topic.
After the completion of this Cloud Lab, you’ll be able to create MSK clusters and configure the brokers it launches according to your requirements. You’ll also be able to create Kafka topics and add producers and consumers to them.
The following is the high-level architecture diagram of the infrastructure you’ll create in this Cloud Lab:
Relevant Course
Use the following content to review prerequisites or explore specific concepts in detail.
Frequently Asked Questions
What is AWS-managed streaming for Apache Kafka?
Amazon MSK is a fully managed service by AWS that makes it easy to use Apache Kafka for building real-time streaming applications. It takes care of tasks like setting up, configuring, and maintaining Kafka clusters so you can focus on your applications without worrying about managing the infrastructure.
Does Amazon use Apache Kafka?
Amazon supports Apache Kafka through its fully managed service, Amazon MSK. This service runs open-source Apache Kafka, ensuring compatibility with existing applications and tools without requiring code changes.
What is the AWS streaming service like Kafka?
The AWS streaming service similar to Kafka is Amazon Managed Streaming for Apache Kafka (Amazon MSK). It is a fully managed service that makes running Apache Kafka on AWS easy without managing infrastructure. Besides this, Amazon Kinesis Data Streams is a fully managed service that enables real-time processing and analysis of data streams, making it easy to build applications that seamlessly handle live data. However, it is not Kafka-compatible.
Is MSK the same as Kafka?
No, Amazon MSK is not the same as Apache Kafka. MSK is a managed service that runs Apache Kafka for you and allows you to create and manage Kafka clusters without dealing with manual setup and maintenance complexities.
What is the difference between Apache Kafka and Kafka streams?
Apache Kafka is a distributed event streaming platform for storing, publishing, and subscribing to real-time messages. On the other hand, Kafka Streams is a library that lets developers process and analyze data streams from Kafka. Simply put, Kafka Streams is a complementary tool that adds stream processing capabilities to your application that consumes data from Apache Kafka.
Shamir Alavi
Felipe Matheus
Software Engineer
Adina Ong
Senior Engineering Manager
