Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

The Apache Spark File Format Ecosystem - Optimizing Storage for Performance

Databricks via YouTube

Overview

Explore the Apache Spark file format ecosystem in this 29-minute video from Databricks. Dive into the critical role of storage and IO in Spark job performance and optimization. Learn about key file formats like Parquet, ORC, and Avro, their origins, core data structures, and optimization techniques. Discover how to tune SparkConf and SQLConf settings for these formats. Examine real-world industry examples showcasing file format impact on job performance and stability. Gain insights into emerging technologies like Apache Arrow. By the end, understand core concepts of prevalent file formats, relevant settings, and how to select the right format for your Spark jobs, especially for IO-intensive AI/ML workflows.

Syllabus

Intro
Session Goals
File Formats
Row-wise Storage
Columnar (Column-wise) Storage
Hybrid Storage
Example Data
About: CSV
About: JSON
About: Avro
Inspecting: Avro
About: ORC
Structure: ORC
Inspecting: ORC
Config: ORC
Structure: Parquet
Inspecting: Parquet (1)
Inspecting: Parquet (2)
Config: Parquet
Case Study: Veraset
Looking Forward: Apache Arrow
Final Thoughts

Taught by

Databricks

Reviews

Start your review of The Apache Spark File Format Ecosystem - Optimizing Storage for Performance

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.