Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Degrading Performance - Understanding and Solving Small Files Syndrome

Databricks via YouTube

Overview

Explore the impact of small files on data pipeline performance in this 27-minute video from Databricks. Learn about the challenges posed by small files, particularly in Parquet format, across real-time, near-real-time, and batch processing jobs. Understand how cloud storage interacts with Apache Spark, delve into Parquet and Storage API calls, and discover why small files can degrade system performance. Gain insights into leveraging DeltaLake for a more efficient solution. Follow along with a comprehensive agenda covering Spark query lifecycle abstraction, Azure Data Lake Gen 2, file hierarchy detection, and a practical demonstration.

Syllabus

Intro
Agenda
Spark query lifecycle abstraction
Azure Data Lake Gen 2
Detecting
File Hierarchy
Demo
Demo Summary
Outro

Taught by

Databricks

Reviews

Start your review of Degrading Performance - Understanding and Solving Small Files Syndrome

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.