Downscaling Apache Spark Clusters - Challenges and Solutions

Explore the challenges and solutions for downscaling Apache Spark clusters in this 36-minute conference talk by Prakhar Jain from Databricks. Dive into the complexities of removing nodes from running Spark-on-Yarn clusters when workload decreases, addressing issues like container fragmentation and shuffle data retention. Learn about innovative approaches to improve downscaling, including changes in YARN's container allocation strategy and Spark's task scheduler for better container packing. Discover enhancements to Spark driver and External Shuffle Service (ESS) that enable proactive deletion of consumed shuffle data, facilitating faster node reclamation. Gain insights into terminology, resource allocation strategies, and the impact of minimum executors on downscaling. Examine the production and consumption of shuffle data, the role of ESS, and potential solutions for long-running applications. Conclude with an overview of Spark's compute and storage disaggregation and future directions for cluster downscaling optimization.