Chasing the Stream Processing Utopia

Explore the challenges and advancements in real-time data processing at internet scale in this conference talk from Strange Loop. Dive into the world of stream processing as Kartik Paramasivam shares LinkedIn's experience handling trillions of events daily using Apache Kafka and Samza. Learn about the evolution from batch to stream processing, the importance of real-time data reactions in social computing, and the current state of stream processing technology. Discover techniques for efficient event processing, including application local state for improved performance, and the need for multiple programming languages in expressing processing logic. Gain insights into scaling ingestion, managing Kafka clusters, and addressing bottlenecks in stream processing. Examine concepts such as stateless and stateful applications, windowing, event time-based processing, and exactly once processing. Understand the challenges of maintaining temporary state, working with batch sources, and the tools ecosystem surrounding stream processing technologies.