Migrating ETL Workflows to Apache Spark at Scale - Pinterest's Experience

Explore Pinterest's journey in migrating their batch processing to Apache Spark in this 25-minute conference talk from Databricks. Discover the challenges and solutions encountered during the transition from legacy ETL workflows written in Cascading/Scalding. Learn about the migration's motivation, bridging semantic gaps between different engines, handling thrift objects, improving Spark accumulators, and performance tuning using an innovative Spark profiler. Gain insights into the performance improvements and cost savings achieved post-migration. Delve into topics such as Spark clusters, API approaches, translating Cascading and Scalding, secondary sort, accumulator enhancements, profiling techniques, automatic migration services, data validation, and balancing performance. Understand the complexities of large-scale ETL workflow migration and the future plans for Pinterest's data processing infrastructure.