Second Generation Workflow Managers for Big Data

Explore the evolution of big data workflow management in this conference talk. Discover how second-generation tools are revolutionizing job scheduling and workflow management for large-scale data processing. Learn about the shift from XML and DSL-based descriptions to Python-based solutions, offering greater versatility and agility. Examine Luigi, an open-source workflow manager developed by Spotify, and gain insights into its features and capabilities. Compare Luigi with other emerging tools like Pinterest's Pinball and Airbnb's Airflow, understanding their unique strengths and applications in the rapidly evolving big data landscape.