Improving Broadcast Joins in Apache Spark SQL

Explore the intricacies of broadcast joins in Apache Spark SQL through this 28-minute Databricks conference talk. Delve into the mechanics of Spark's execution engine, focusing on broadcast joins and their performance implications. Learn about Workday's improvements to increase the threshold for effective broadcast joins, including executor-side broadcasting and modifications to Spark's whole-stage code generator. Discover techniques for limiting memory usage in executors while increasing broadcasting thresholds. Gain insights from real-world production case studies involving large-scale ETL pipelines. Acquire valuable knowledge to optimize your own Spark workloads and enhance your understanding of Spark's join infrastructure.