Downscaling Apache Spark Clusters - Challenges and Solutions

Downscaling Apache Spark Clusters - Challenges and Solutions

Databricks via YouTube Direct link

Factors affecting node downscaling

5 of 21

5 of 21

Factors affecting node downscaling

Class Central Classrooms beta

YouTube videos curated by Class Central.

Classroom Contents

Downscaling Apache Spark Clusters - Challenges and Solutions

Automatically move to the next video in the Classroom when playback concludes

  1. 1 Intro
  2. 2 Autoscaling on cloud
  3. 3 Upscale easy, downscale difficult
  4. 4 How are nodes used?
  5. 5 Factors affecting node downscaling
  6. 6 Terminology Any cluster generally comprises of following entities: • Resource Manager
  7. 7 Current resource allocation strategy
  8. 8 Example revisited with new allocation strategy
  9. 9 Downscale issues with Min Executors
  10. 10 Min executors distribution without packing
  11. 11 Min executors distribution with packing
  12. 12 How Shuffle data is produced / consumed?
  13. 13 External Shuffle Service
  14. 14 ESS at Qubole
  15. 15 Recap
  16. 16 Shuffle Cleanup • Shuffle data is deleted at the end of application by ESS
  17. 17 Issues with long running applications
  18. 18 Shuffle reuse in Spark
  19. 19 Downscaling a Node
  20. 20 Spark - Disaggregation of Compute and Storage • Mount some NFS endpoint on all the nodes of cluster • Change shuffle manager in Spark to something which can read/write shuffle from NFS mount point
  21. 21 Summary and Future Work

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.