Building a Batch Processing Platform for Data Pipelines Using Argo and Kubernetes

Explore a conference talk detailing Intuit's development of a highly scalable batch processing platform using Kubernetes and Argo for efficient data pipeline management. Discover how this solution addresses challenges in scheduling, orchestration, and complex dependency management for over 100,000 data pipelines across hundreds of AI and Data engineering teams. Learn about the integration of Argo Events, Argo Workflow, and Kubernetes to create an effective orchestration and scheduling engine for various data processing use cases. Gain insights into the operational challenges of managing multi-cluster Kubernetes infrastructure and the integration of Argo with Kafka for zero downtime scheduling. Understand how this holistic approach eliminates silos and enhances processing effectiveness in the data lake environment.