Democratization of ML Pipelines and Bringing ML Workflows to Heterogeneous Cloud Native Platforms

Explore the democratization of machine learning pipelines and the adaptation of ML workflows to heterogeneous cloud-native platforms in this informative conference talk. Delve into the evolution of Kubeflow Pipelines (KFP) from v1 to v2, focusing on the introduction of Intermediate Representation (IR) as a platform-independent pipeline specification. Learn about the new pipeline orchestration engine that supports automatic lineage tracking and metadata-driven components. Discover how IR enhances ML pipeline portability, allowing for execution on various desired platforms. Examine the use of IR in the Kubeflow pipeline registry protocol as a standardized format for pipeline templates. Gain insights into the stable IR specification adopted by Kubeflow Pipelines, Google Vertex AI, and Kubeflow Pipelines with Tekton. Follow along as the speakers walk through the new IR spec, components of the pipeline orchestration engine, adaptation strategies for other pipeline frameworks, and the implementation of the new registry protocol to democratize ML pipeline development.