Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

CNCF [Cloud Native Computing Foundation]

Building Scalable and Effective Data Pipelines with Argo for Large-Scale Metadata Processing

CNCF [Cloud Native Computing Foundation] via YouTube

Overview

Watch a 24-minute conference talk exploring how Atlan's data team revolutionized their ETL pipeline performance using Argo workflows. Learn how they overcame challenges of massive data processing by implementing parallelization techniques, retry mechanisms, and effective artifactory loading. Discover the technical journey that led to processing 100 million assets with a 450% reduction in processing time and handling 3 million SQL queries in just 2 hours. Gain insights into scaling data pipelines effectively as speakers detail how they achieved a 50% reduction in overall pipeline runtime through Argo-powered horizontal scaling solutions.

Syllabus

Argonauts of Data: Building Scalable and Effective Data Pipelin... Satabrata Paul & Nishchith Shetty

Taught by

CNCF [Cloud Native Computing Foundation]

Reviews

Start your review of Building Scalable and Effective Data Pipelines with Argo for Large-Scale Metadata Processing

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.