Serverless Realtime Lakehouse Architecture - Standing on the Shoulders of Open-Source Giants

Explore the evolution and implementation of the lakehouse architecture in this 49-minute conference talk from Confluent. Gain a technical, no-nonsense introduction to the lakehouse concept, delve into its architecture, and understand how it's built with streaming as a core component. Discover the potential of serverless technology for streaming use cases, drawing parallels with popular serverless concepts like AWS Lambda functions. Participate in an interactive, hands-on demo showcasing a serverless streaming use case involving live event ingestion from hundreds of mobile devices. Critically examine the extent of serverless implementation in modern lakehouses, with insights from Databricks' approach. Recap the open-source foundation of lakehouse architecture, understand its appeal for streaming beyond Spark Structured Streaming, and explore the technical aspects of a serverless lakehouse. Learn how this architecture can benefit data practitioners by reducing costs and operational overhead through a live, end-to-end demonstration of serverless data engineering.