Building a Management Layer for Structured and Unstructured Data Lakes

Learn how to effectively manage data lakes for both structured and unstructured data in this comprehensive technical talk. Explore the key architectural components including open table formats, catalogs, and data version control systems while understanding their crucial roles in data lake management. Examine practical implementations through detailed examples using Databricks technologies, Apache Iceberg, and AWS solutions. Gain insights from industry expert Einat Orr, CEO and Co-founder of Treeverse, as she draws from her extensive experience in engineering leadership and academic background in mathematics to address common challenges and best practices in data lake management. Master the integration of various components to create a robust management layer that enhances data lake functionality and maintainability.