Transforming Data Processing with Kubernetes - Journey Towards a Self-Serve Data Mesh

Explore how Intuit leverages Kubernetes with Domain-Driven Design and Data Mesh principles to revolutionize its data processing landscape in this 34-minute conference talk. Discover the critical role of clean data in developing robust generative artificial intelligence and learn about Intuit's approach to creating paved paths for data platforms on Kubernetes. Examine the challenges and solutions in managing 100,000 data pipelines and coordinating 1000+ engineers interacting with data, emphasizing the need for scalable solutions. Gain insights into Intuit's use of Kubernetes to build its batch and stream processing platform, addressing hurdles in data pipeline deployment, scheduling, orchestration, and dependency management. Understand how this transformation, rooted in treating data as a product, has enhanced decision-making speed and accuracy across the organization while fostering a more efficient, collaborative data culture.