Bringing ML Workflows to Heterogeneous Cloud Native Machine Learning Platforms Using Intermediate Representation

Explore the evolution of machine learning workflows in cloud-native environments through this 40-minute conference talk by Yi-Hong Wang from IBM. Delve into the introduction of Intermediate Representation (IR) in Kubeflow Pipelines v2, designed to enhance ML pipeline portability across different frameworks. Learn about the new pipeline orchestration engine that supports automatic lineage tracking and metadata-driven components. Discover how IR specification is utilized by Kubeflow Pipelines, Google Vertex AI, and Kubeflow Pipelines with Tekton. Gain insights into the new IR spec, components of the pipeline orchestration engine, and the adaptation of these features in other pipeline frameworks. Understand how to leverage the Kubeflow Pipelines Python SDK to convert ML flows into IR artifacts and execute them on multiple orchestration engines. The talk covers key topics including AI lifecycle pillars, pipeline definition with Python SDK, benefits of metadata and artifact tracking, Machine Learning Metadata in v1, Pipeline Spec in v1, Smart Runtime, and the Abstraction Layer for Orchestration Engines with its features and benefits.