Streamlined Monitoring of Databricks Workflows with Deloitte

Discover how Deloitte leverages the Databricks Data Intelligence Platform to create an Automated Workflow Monitoring solution in this 17-minute conference talk. Learn about a configurable and automated approach to track Lakehouse data engineering tasks with minimal overhead costs in a single simplified view. Explore how this solution, built on Databricks' SDK and SQL dashboards, creates a job to monitor all data engineering pipeline statuses in a Databricks Workspace. Understand how it automates large parts of operations and maintenance, reduces overhead costs, and provides trustworthy insights for critical data pipeline tasks. Gain insights into building configurations that meet enterprise needs and developing extensions such as customized email alerts or connections to business intelligence tools. Presented by Ashvic Godinho, Data Engineering Technical Lead at Deloitte, this talk offers valuable knowledge for data professionals seeking to streamline their Databricks workflow monitoring processes.