Streaming DataFrames: A New Way to Process Streaming Data - Conf42 Python 2024

Overview

Explore a new approach to processing streaming data in this conference talk from Conf42 Python 2024. Discover the challenges of traditional stream processing methods and learn about Streaming DataFrames as an innovative solution. Gain insights into stream processing architectures, Kafka producer and consumer APIs, and various frameworks. Understand the limitations of current approaches, including debugging difficulties and UDF complexities. Delve into the speaker's racing background and how it relates to real-time data processing. Watch a live demonstration of Quix Streams, a tool designed to simplify stream processing using Streaming DataFrames. Learn about the goals and benefits of this new methodology, and find out how to try Quix for yourself.

Syllabus

intro
preamble
hello, nice to meet you!
racing background
agenda
why strean processing?
stream processing architecture
kafka producer and consumer apis
stream processing frameworks
stream processing library example
debugging
jar files...
udfs are nasty
our approach to stream processing
goals
is there a third way?
quix streams - sdf
goal
demo
try quix