Data Discovery at Databricks with Amundsen - Improving Productivity and Trust

Explore how Databricks leverages Amundsen, an open-source data discovery tool, to enhance productivity and trust in internal data exploration. Learn about the integration of Amundsen with Databricks' infrastructure to surface metadata, including popular tables, fuzzy and facet search capabilities, and rich dataset information such as lineage, ownership, and usage statistics. Discover how the tool provides insights on ETL jobs, column statistics, and associated dashboards. Gain knowledge about the implementation of user feedback and plans to extend these discovery improvements to Databricks customers. Delve into the deployment details, development process, and specific metadata surfaced in Amundsen, including table lineage generation, Delta table extended metadata, and Redash dashboard integration. Understand the benefits of this data discovery solution compared to the previous static wiki approach and its potential impact on data-driven decision-making within the organization.