Delta Lake for Polyglot Data and Machine Learning Workloads

Explore the power of Delta Lake in this 18-minute conference talk by Micha Kunze, Lead Data Engineer at Maersk. Discover how Delta Lake serves as a robust open table format for managing over 1000 datasets with 20,000+ daily job runs. Learn about its seamless integration with Apache Spark™, providing exactly-once semantics for micro-batching data transport and transformation. Delve into Delta Lake's versatility, supporting Python and Rust for machine learning applications. Gain insights into leveraging Delta as the backbone for operational data streams, explorative data analysis, and reproducible ML workloads. Understand techniques for monitoring data quality and streams using table metadata. Explore practical applications of Delta, including implementing WAP patterns to prevent bad data, optimizing compute job efficiency, building auto-ML models, and automating Spark Structured Streaming job monitoring and alerting.