Scaling Kubernetes: Best Practices for Managing Large-Scale Batch Jobs with Spark and Argo Workflow

Discover best practices for managing large-scale batch jobs on Kubernetes with Spark and Argo Workflow in this informative conference talk. Learn how to build a Kubernetes cluster capable of frequently creating and deleting 20,000 pods for parallel computation. Explore techniques for optimizing the Kubernetes control plane, including list-watch mechanism, service broadcasting, environment variable attachments, and API server configurations. Gain insights into configuring Spark operator and Argo workflows controller for improved performance. Perfect for those managing data processing with Spark applications or genomics computing with Argo workflows, this presentation offers valuable guidance on scaling Kubernetes to handle significant workloads efficiently.