Understanding and Improving Code Generation in Spark

Explore the intricacies of code generation in Spark's physical execution engine in this 24-minute conference talk. Dive into the differences between expression codegen and whole-stage codegen, and learn how Workday has improved code generation to handle complex queries. Discover the challenges posed by large generated functions, including OOM errors, Java method size limitations, and performance regressions. Understand the innovative approach to splitting collapsed functions from whole-stage codegen while maintaining performance benefits. Gain insights into the performance improvements achieved in production workloads through these enhancements. Follow the journey from the Volcano Iterator Model to Stage Cogeneration, examining the problems encountered and the solutions implemented to optimize Spark's code generation capabilities.