Explore a comprehensive talk on Delta Live Tables (DLT), a revolutionary ETL framework that simplifies data transformation and pipeline management. Learn how DLT incorporates modern software engineering practices to deliver reliable and trusted data pipelines at scale. Discover techniques for rapid innovation in pipeline development and maintenance, automation of administrative tasks, and improved visibility into pipeline operations. Gain insights into built-in quality controls and monitoring for accurate BI, data science, and ML. Understand how to implement simplified batch and streaming with self-optimizing and auto-scaling data pipelines. Delve into topics such as live table dependencies, pipeline boundaries, SQL expectations, Python integration, metaprogramming, event log integration, failure recovery automation, Spark Structured Streaming for ingestion, and Delta for infinite retention.
Overview
Syllabus
Intro
Life as a data professional.
What is a Live Table?
Development vs Production
Declare LIVE Dependencies
Choosing pipeline boundaries
Pitfall: hard-code sources & destinations
Ensure correctness with Expectations
Expectations using the power of SQL
Using Python
Installing libraries with pip
Metaprogramming in python
Best Practice: Integrate using the event log
DLT Automates Failure Recovery
What is SparkTM Structured Streaming?
Using Spark Structured Streaming for ingestion
Use Delta for infinite retention
Partition recomputation
Taught by
Databricks
Related Courses
-
Declarative ETL Pipelines with Delta Live Tables - Modern Software Engineering for Data Analysts and Engineers
-
Data Engineering with Databricks
-
Delta Live Tables: Building Reliable Data Pipelines with Databricks
-
Delta Live Tables - Building Reliable ETL Pipelines with Azure Databricks
-
Delta Live Tables in Depth - Best Practices for Intelligent Data Pipelines
-
Efficient Near Real-Time Event Ingestion using DLT - Insights and Lessons
