Towards Multi-Table Transactions in Delta Lake

Explore the evolution of Delta Lake's commit protocol in this 20-minute conference talk. Delve into the concept of managed-commits, a new approach that shifts commit atomicity from object stores to external commit owners like HMS, Unity Catalog, or Glue. Discover how this change lays the foundation for advanced features such as multi-statement transactions and addresses limitations in cloud storage primitives. Learn about the benefits of managed-commits, including support for multi-table-multi-statement transactions, reliable commit semantics for object stores lacking put-if-absent capabilities, and improved data governance operations. Gain insights from Prakhar Jain, Staff Software Engineer at Databricks, as he discusses the technical details and implications of this new commit protocol for Delta Lake.