Optimizing Catalyst Optimizer for Complex Spark Plans

Optimizing Catalyst Optimizer for Complex Spark Plans

Databricks via YouTube Direct link

Intro

1 of 22

1 of 22

Intro

Class Central Classrooms beta

YouTube videos curated by Class Central.

Classroom Contents

Optimizing Catalyst Optimizer for Complex Spark Plans

Automatically move to the next video in the Classroom when playback concludes

  1. 1 Intro
  2. 2 Spark in Workday Prism Analytics
  3. 3 Example: Data Validation
  4. 4 About Complex Plans
  5. 5 Common Subexpression Elimination (CSE)
  6. 6 CSE Benchmark
  7. 7 Logging Complex Plans (10s of MBs in Size)
  8. 8 Problems with Large Case Expressions
  9. 9 Handling Large Case Expressions in Catalyst
  10. 10 Large Case Expression Benchmark
  11. 11 Example: Generate New Filter
  12. 12 Example: Prune Redundant Filter
  13. 13 Example: New Filter on Other Side of Join
  14. 14 Current Constraint Propagation Algorithm
  15. 15 Current Algorithm Takes High Memory
  16. 16 Recall: Fix for Large Case Expressions
  17. 17 Optimized Constraint Propagation (SPARK-33152)
  18. 18 Constraint Propagation Algorithms Comparison
  19. 19 Constraint Propagation Benchmark
  20. 20 Effect on Customer Pipeline
  21. 21 Tuning Tips
  22. 22 Future Work

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.