Through the Looking Glass: Key Architectural Choices in Flink and Kafka Streams

Explore a comparative analysis of Apache Flink and Kafka Streams in this 43-minute conference talk. Delve into fundamental stream processing concepts and examine the architectural choices that differentiate these two battle-tested production systems. Gain insights into topic-based shuffle, key-groups primitives, state partition management, end-to-end exactly-once semantics, and changelog implementations. Learn how these design decisions impact system performance, failover capabilities, scalability, and multi-cluster support. Discover practical implications for your own stream processing projects and walk away with a deeper understanding of stream processing fundamentals. Presented by John Roesler, principal software engineer at Confluent and Apache Kafka PMC member, this talk offers valuable knowledge for both experienced developers and those new to stream processing technologies.