Building Scalable and Effective Data Pipelines with Argo for Large-Scale Metadata Processing
CNCF [Cloud Native Computing Foundation] via YouTube
Overview
Watch a 24-minute conference talk exploring how Atlan's data team revolutionized their ETL pipeline performance using Argo workflows. Learn how they overcame challenges of massive data processing by implementing parallelization techniques, retry mechanisms, and effective artifactory loading. Discover the technical journey that led to processing 100 million assets with a 450% reduction in processing time and handling 3 million SQL queries in just 2 hours. Gain insights into scaling data pipelines effectively as speakers detail how they achieved a 50% reduction in overall pipeline runtime through Argo-powered horizontal scaling solutions.
Syllabus
Argonauts of Data: Building Scalable and Effective Data Pipelin... Satabrata Paul & Nishchith Shetty
Taught by
CNCF [Cloud Native Computing Foundation]