How We Used Kafka to Scale Database Infrastructure

Explore a comprehensive conference talk from SREcon19 Asia/Pacific that delves into scaling database infrastructure using Kafka. Learn about LinkedIn's home-grown NoSQL document store, its initial MySQL-based design, and the challenges faced. Discover the major changes implemented to replace MySQL replication with Kafka-based replication, including in-depth coverage of the implementation. Gain insights into Kafka configuration options for reliable delivery and application logic ensuring "exactly once delivery" and "rejection of out-of-band messages." Understand the workings of the NoSQL datastore, challenges with MySQL-based replication, how Kafka helped scale the infrastructure, and overall strategies for using Kafka for database replication at scale. The talk covers topics such as Espresso's features, current scale, basic architecture, replication requirements, local and cross-colo replication, cluster expansion, implementing Kafka-based replication, global transaction identifiers, replication flow, message protocols, checkpointing, and handling producer failures.