Exactly-Once Semantics Revisited: Distributed Transactions Across Flink and Kafka

Explore the challenges and upcoming improvements in Exactly-Once Semantics (EOS) integration between Apache Flink and Apache Kafka in this 31-minute talk. Dive into the current pitfalls of Flink's EOS implementation when writing to Kafka, including the use of Java reflection hacks and potential data loss scenarios. Learn about the collaborative efforts of the Flink and Kafka communities to address long-standing technical debts. Gain insights into the basics of Flink's EOS achievement with external systems and common hurdles in implementing distributed transactions. Discover the proposed changes to both the Kafka transaction protocol and Flink transaction coordination aimed at creating a more robust integration. By the end of this session, understand the unique challenges of EOS with Flink and Kafka, and get a glimpse of the anticipated improvements across both projects.