Optimizing Kafka Consumers with Kubernetes and KEDA - Lessons from the Trenches

Explore strategies for optimizing Kafka consumer services using Kubernetes and KEDA in this informative conference talk. Learn how to enhance scalability and efficiency while maintaining service level agreements (SLAs) in real-time data processing. Discover the journey from using Horizontal Pod Autoscaler (HPA) with CPU and memory metrics to implementing KEDA's custom metrics for Kafka consumer lag. Gain insights into achieving over 75% efficiency while honoring data freshness SLAs. Delve into the implementation of a Custom Controller Resource Definition (CRD) for dynamic vertical scaling in Kafka topics, complementing KEDA's handling of daily traffic fluctuations. Understand the benefits of this holistic approach and valuable lessons learned in the pursuit of peak efficiency for Kafka consumers in a cloud-native environment.