Care and Feeding of Catalyst Optimizer - Practical Troubleshooting for Spark SQL

Explore the intricacies of Spark's Catalyst query optimizer in this 42-minute talk from Databricks. Delve into practical challenges and edge cases that arise when working with Spark SQL, focusing on diagnosing and solving issues that require a deep understanding of Spark internals. Learn how to handle scenarios where UDFs unexpectedly become expensive, causing skew. Discover techniques for addressing codegen stage issues that exceed 64k method limits. Gain insights into improving Spark application performance by tuning the JVM code cache. Through a series of puzzles and real-world examples, develop a more sophisticated understanding of Spark's Catalyst Optimizer and enhance your ability to troubleshoot and optimize complex queries.