Hydroflow: A Compiler Target for Fast, Correct Distributed Programs

Explore a groundbreaking compiler stack designed to enhance the correctness and efficiency of distributed programs in this OOPSLA 2023 conference talk. Delve into Hydroflow, a Rust-based dataflow runtime that forms the foundation of the Hydro project at Berkeley. Discover how Hydroflow's semilattice formalism-based IR enables natural program transformations in distributed systems, including refactoring, replication, and partitioning. Learn about the optimization potential for distributed programs, addressing parallelism, memory scaling, performance isolation, geoproximity, and physical security. Examine case studies demonstrating correctness, latency, and scaling results when optimizing various programs, from key-value stores to shopping carts and complex consensus protocols.