ExoFlow - A Universal Workflow System for Exactly-Once DAGs

Explore a groundbreaking universal workflow system in this conference talk from OSDI '23. Dive into ExoFlow, a system designed to provide flexible recovery and performance tradeoffs within the same application. Learn how ExoFlow decouples execution from recovery, offering exactly-once semantics as a separate layer. Discover the innovative use of task annotations to specify execution semantics, enabling the workflow system and end-users to control recovery processes. Understand how ExoFlow generalizes recovery for various workflow applications, from ETL pipelines to stateful serverless workflows, while introducing optimizations in task communication and recovery. Gain insights into the challenges of implementing interoperable recovery techniques in heterogeneous pipelines and how ExoFlow addresses these issues.