Approaching Kafka Replication at Scale - Go Big or Go Home

Explore the challenges and solutions of scaling Apache Kafka replication in this 32-minute conference talk from Confluent. Learn how New Relic tackled the complexities of breaking down their massive Kafka cluster into smaller, more manageable units. Discover techniques for routing events between clusters and topics based on headers, and understand the journey from fan-out replication to many-to-one and many-to-many replication. Gain insights into overcoming bottlenecks encountered during scaling, including replicating data based on Kafka headers, connecting to multiple source and destination clusters, managing replication of varying traffic topics, and utilizing an intermediary Kafka cluster. By the end of this talk, grasp how New Relic scaled their replication and routing to support growing data ingestion needs and the various mitigations implemented along the way.