Explore stream processing system architecture in this technical talk from RisingWave Labs CEO Yingjun Wu at the Dutch Seminar on Data Systems Design. Dive deep into the benefits and challenges of decoupling compute and storage in stream processing systems, with particular focus on addressing data consistency and latency issues when managing large-scale internal states. Learn about an innovative tiered storage mechanism that combines high-performance and low-cost storage tiers to optimize data movement between compute and storage layers. Examine experimental results demonstrating how this approach achieves an optimal balance between performance and cost-efficiency. Drawing from over a decade of experience in stream processing and database systems, including work at Amazon Web Services Redshift team and IBM Almaden Research Center, Wu shares practical insights into building scalable stream processing infrastructure.
Decoupling Compute and Storage for Stream Processing Systems
DSDSD - Dutch Seminar on Data Systems Design via YouTube
Overview
Syllabus
We hold bi-weekly talks on Fridays from PM to 5 PM CET for and by researchers and practitioners designing and implementing data systems. The objective is to establish a new forum for the Dutch Data Systems community to come together, foster collaborations between its members, and bring in high-quality international speakers. We would like to invite all researchers, especially also Ph.D. students, who are working on related topics to join the events. It is an excellent opportunity to receive feedback early on from researchers in your field.
Taught by
DSDSD - Dutch Seminar on Data Systems Design
Related Courses
-
LingoDB: An Open Compilation and Optimization Framework for Sustainable Data Processing
-
Leveraging Generative AI for Data Processing and Database Management
-
Shredding Deeply Nested JSON for Efficient Analytics in DuckDB
-
Parameter-Efficient Automation of Data Wrangling Tasks with Prefix-Tuning
-
Provenance Research in Data Management Systems at Microsoft
-
Implementing InfluxDB IOx Using Apache Arrow, DataFusion, and Rust
