Explore a new approach to processing streaming data in this conference talk from Conf42 Python 2024. Discover the challenges of traditional stream processing methods and learn about Streaming DataFrames as an innovative solution. Gain insights into stream processing architectures, Kafka producer and consumer APIs, and various frameworks. Understand the limitations of current approaches, including debugging difficulties and UDF complexities. Delve into the speaker's racing background and how it relates to real-time data processing. Watch a live demonstration of Quix Streams, a tool designed to simplify stream processing using Streaming DataFrames. Learn about the goals and benefits of this new methodology, and find out how to try Quix for yourself.
Streaming DataFrames: A New Way to Process Streaming Data - Conf42 Python 2024
Overview
Syllabus
intro
preamble
hello, nice to meet you!
racing background
agenda
why strean processing?
stream processing architecture
kafka producer and consumer apis
stream processing frameworks
stream processing library example
debugging
jar files...
udfs are nasty
our approach to stream processing
goals
is there a third way?
quix streams - sdf
goal
demo
try quix
Taught by
Conf42
Related Courses
-
Processing Streaming Data Using Apache Flink
-
A Deep Dive into Apache Kafka - This is Event Streaming
-
Kafka as a Platform - The Ecosystem from the Ground Up
-
Apache NiFi, Kafka, RisingWave, and Iceberg for Stock Data Processing with LLM - Conf42 Python 2024
-
Apache Kafka Tutorials
-
Apache Kafka and KSQL in Action - Let’s Build a Streaming Data Pipeline
