Bringing ML Workflows to Heterogeneous Cloud Native Machine Learning Platforms Using Intermediate Representation

Explore the evolution of machine learning workflows in cloud-native environments through this conference talk. Delve into the concept of Intermediate Representation (IR) in Kubeflow Pipelines v2, designed to enhance ML pipeline portability across different frameworks. Learn about the new pipeline orchestration engine that supports automatic lineage tracking and metadata-driven components. Discover how IR facilitates deployment of ML pipelines on various platforms, including Kubeflow Pipelines and Google Vertex AI. Gain insights into the IR specification, components of the new pipeline orchestration engine, and its adaptation to other pipeline frameworks. Examine the pillars of AI lifecycle, pipeline definition using Python SDK, and the benefits of metadata and artifact tracking. Understand the transition from Kubeflow Pipelines v1 to v2, exploring improvements in Machine Learning Metadata and pipeline specifications. Investigate the abstraction layer for orchestration engines, its benefits, and features such as execution client and execution spec. Acquire knowledge about Argo Workflows, Kubeflow Pipelines with Tekton, and TensorFlow Extended's use of MLMD as a metadata store.