Netflix Maestro: Orchestrating Scaled Data and ML Workflows in the Cloud

Watch a 23-minute conference talk from AWS re:Invent 2023 exploring Netflix's development of Maestro, their innovative workflow orchestrator designed for managing large-scale data and machine learning operations in the cloud. Discover how Netflix addresses the growing demands of big data and ML by creating a system that enhances scalability, reliability, and usability. Learn about Maestro's capabilities in handling complex workflows with hundreds of thousands of nested jobs while maintaining comprehensive lineage information between event signals, workflows, and tables. Explore the technical architecture, critical design considerations, and practical implementations through detailed explanations of interfaces, integrations, DSL, backfill workflows, and execution abstractions. Gain insights into why Netflix chose to build their own solution and how it has improved engineer productivity across their data ecosystem.