Explore the process of building scalable data engineering pipelines using Delta Lake in this 38-minute conference talk by Amanda Moran from Databricks. Learn about the 'multi-hop' architecture, which uses Bronze, Silver, and Gold tables to progressively structure data from ingestion to machine learning. Discover how to implement this architecture using Delta Lake, enabling a single source of truth for raw data. Follow along with a live demo showcasing importing data, creating Bronze and Silver tables, performing updates, deletes, and merges, as well as managing schema evolution. Gain insights into the Delta Lake lifestyle and its community, empowering you to become a champion in your organization's data engineering efforts.
Overview
Syllabus
Intro
Amandas background
Agenda
Data Engineers Journey
Delta Architecture
Delta Lake Architecture
Data Lifecycle Analogy
The Delta Lake Lifestyle
What can we do with Delta
Whats in the notebook
Importing data
Creating a bronze table
Creating a silver table
Creating a silver Delta table
Description of the silver Delta table
Live Demo
Updates Deletes and merges
Merges
Schema Evolution
Describe History
Recap
Using Delta Lake
Delta Lake Community
Taught by
Databricks
