Breaking the 1.5MB Barrier: Running Large Metaflow Flows with Argo for AI/ML Workloads

Learn how to overcome etcd's 1.5MB object size limitation when running large-scale AI/ML workflows in Argo through this technical conference talk. Discover the challenges of managing extensive batch workflows for AI/ML workloads, particularly in data preparation and model training scenarios that involve numerous steps. Explore real-world examples demonstrating how this limitation creates unpredictable bottlenecks, and master the implementation of Argo's innovative solution using RDBMS offloading. Follow a detailed walkthrough of configuring the Argo offloading feature with Aurora Postgres RDS and EKS in AWS environments, enabling seamless scaling while maintaining Kubernetes compatibility.