Monitoring ML Models, IoT Data, and Quality Checks on Delta Lake - Quby's Approach

Explore a comprehensive 30-minute presentation on monitoring large-scale machine learning models, IoT streaming data, and automated quality checks using Delta Lake. Dive into Quby's innovative approach to managing Europe's largest energy dataset, consisting of petabytes of IoT data. Learn how Delta Lake ensures data quality through schema enforcement and evolution, and discover the crucial role of Data Engineers in verifying timely data ingestion with expected metrics. Examine the challenges of training and serving over half a million models daily, and understand the importance of balancing quality data with well-performing models. Gain insights into monitoring raw and processed data quality metrics using Databricks dashboards, tracking model performance with MLflow, and implementing Slack alerts for failure notifications. Explore real-world examples of managing large-scale data processing and machine learning pipelines in production environments.