Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Boost Delta Lake Performance with Data Skipping and Z-Order

Databricks via YouTube

Overview

Explore advanced techniques for optimizing Delta Lake performance in this 41-minute tech talk from Databricks. Learn how to leverage Data Skipping and Z-Ordering to enhance read and write performance in your data lake. Discover the evolving partitioning strategies employed by Salesforce Engineering when building their Engagement delta lake. Gain insights into overcoming challenges associated with small file generation and suboptimal partitioning. Compare traditional file-based partitioning with partition pruning to the innovative approaches of Data Skipping and Z-Ordering with I/O pruning and file Compaction. Benefit from real-world examples and expert insights shared by Salesforce engineers Zhidong Ke, Yifeng Liu, Aaron Zhang, and Heng Zhang as they discuss their experiences in designing high-performance distributed systems and data processing pipelines.

Syllabus

Intro
Project Overview
Optimize
AutoOptimize
Partition
What is ZOrder
Stats
Results
New Partition Scheme
IO Pruning
Data Set
Deleting Proportions
Deleting Optimizations
After Performance Work
Other Performance Considerations
Questions
Outro

Taught by

Databricks

Reviews

Start your review of Boost Delta Lake Performance with Data Skipping and Z-Order

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.