Modernizing Nubank's Kafka Platform - From 1 Trillion Messages per Month to the Next Decade

Explore how Nubank, a leading digital bank and fintech company, modernized its Kafka platform to handle over 1 trillion messages per month across 50+ clusters in three countries. Learn about the challenges faced with operating Kafka on raw EC2 instances for a decade and the decision to migrate to a Strimzi setup. Discover the architecture and process of live migrating to the new platform, including strategies for minimizing failure impact through a cell-based architecture and the mechanism of topic-by-topic migration. Gain insights into the architectural details developed for successful deployment, the concept of Kafka Flavors for cluster specialization, and the optimization of workload management. Understand how Nubank's Kafka infrastructure processes 30% of Brazil's instant transfer transactions and the importance of scalability, security, and automation in managing such a critical system.