Dive into best practices for fine-tuning and enhancing Apache Spark job performance in this 25-minute video from Databricks. Explore real-world problem-solving techniques and learn how to optimize resources by adjusting parameters such as garbage collector selection, serialization, worker/executor numbers, data partitioning, and Java heap settings. Analyze Spark UI execution DAGs to identify bottlenecks, optimize joins, and manage partition sizes. Discover strategies for handling data skew, utilizing scheduling pools, and implementing fair scheduler. Gain insights into Spark SQL rollup best practices and learn which approaches to avoid for improved performance.
Overview
Syllabus
Intro
Our Setup
Configuring Cluster Test change with
Cache/Persist
Join Optimization
Filter Trick
Salting - Reduce Skew
Things to remember
Fair Scheduling
Serialization
Enable GC Logging
ParallelGC (default)
Takeaways
Taught by
Databricks
