Cloud-Native Apache Spark Scheduling with YuniKorn on Kubernetes

Cloud-Native Apache Spark Scheduling with YuniKorn on Kubernetes

Databricks via YouTube Direct link

Intro

1 of 24

1 of 24

Intro

Class Central Classrooms beta

YouTube videos curated by Class Central.

Classroom Contents

Cloud-Native Apache Spark Scheduling with YuniKorn on Kubernetes

Automatically move to the next video in the Classroom when playback concludes

  1. 1 Intro
  2. 2 Role of Kos in Lyft's Data Landscape
  3. 3 Multi-step creation for a Spark KBs job
  4. 4 Problems of existing Spark K8s infrastructure Complexity of layers of custom Kås controllers to handle the scale of the
  5. 5 Why we need a customized K8s Scheduler
  6. 6 Flavors of Running Spark on KBS
  7. 7 Resource Scheduling in K8s
  8. 8 Spark on K8s: the scheduling challenges
  9. 9 Apache Yunikorn (Incubating)
  10. 10 Resource Scheduling in Yunikorn land compare w/default scheduler
  11. 11 Main difference (Yunikorn v.s Default Scheduler)
  12. 12 Run Spark with Yunikorn
  13. 13 Job Ordering
  14. 14 Resource Quota Management: K8s Namespace ResourceQuota
  15. 15 Resource Quota Management: Yunikorn Queue Capacity
  16. 16 Resource Fairness in Yunikorn Queues
  17. 17 Scheduler Throughput Benchmark
  18. 18 Fully K8s Compatible
  19. 19 Yunikorn Management Console
  20. 20 Compare Yunikorn with other K8s schedulers
  21. 21 Current Status
  22. 22 The Community
  23. 23 Roadmap
  24. 24 Our Vision - Resource Mgmt for Big Data & ML

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.