Kafka on Kubernetes - From Evaluation to Production at Intuit

Explore the journey of implementing Kafka on Kubernetes at Intuit in this 30-minute conference talk from USENIX LISA18. Dive into the experimentation and data-driven approach used to evaluate Kafka on Kubernetes in AWS. Learn about the requirements, experimental setup, and various tests conducted, including functional tests for message production and consumption, network isolation, cross-region operations, and performance and scale assessments. Discover the challenges encountered and solutions implemented during the process. Gain insights into the Kubernetes cluster configuration running Kafka, including kafka and zookeeper statefulsets, configmaps for server properties, service objects for broker access, and integration with Splunk and Wavefront for logging and monitoring. Understand the potential of running stateful applications like Kafka on Kubernetes and the considerations involved in moving from evaluation to production in a cloud environment.