Deploying Apache Kafka on DC/OS - Best Practices and Challenges

Overview

Learn best practices, challenges, and lessons learned for deploying Apache Kafka on DC/OS in this 41-minute conference talk by Confluent Solutions Architect Kaufman Ng. Explore the components of a Kafka cluster, understand the benefits of running Kafka on DC/OS, and discover how to manage stateful Kafka services. Dive into topics such as service and broker configuration, storage volumes, Zookeeper integration, broker restarts, and fault-tolerant Kafka Streams. Gain insights on handling failures and familiarize yourself with useful CLI tools for Kafka deployment and management in a DC/OS environment.

Syllabus

Intro
About Confluent
Confluent Platform and Apache Kafka
What is Apache Kafka?
What's in a Kafka Cluster?
Why Kafka on DC/OS?
Kafka Cluster Services
Kafka Services are Stateful
How do you manage all these?
Where to place these services?
DC/OS benefits for Kafka
Service config sample
Broker config sample
Kafka Storage Volumes
What about Zookeeper?
Kafka Broker Restarts
Kafka Streams with fault-tolerance
Failures
Other useful CLI tools