Riding the Stream Processing Wave

Explore the cutting-edge developments in stream processing at LinkedIn in this 36-minute conference talk from Strange Loop. Dive into the three types of applications that have significantly impacted LinkedIn's stream processing platform. Learn about innovative solutions for machine learning applications, including native capabilities like "side-inputs" for handling large state and the ability to convert machine learning DSLs to streaming jobs for feature engineering. Discover the trend of applications transitioning from batch to nearline processing and operating on both batch and streaming datasets. Understand how Samza applications achieve offline-online convergence and the importance of Apache Beam integration. Examine the challenges of maintaining large local states during deployments and failures, and learn about recent improvements in host affinity, state restoration, and the new standby container solution. Gain valuable insights from Samarth Shetty, an experienced engineering leader at LinkedIn, as he shares his expertise in building global-scale cloud infrastructure.