Truly Open Lineage - Building Trust in AI Models Through Data Provenance

Explore the concept of open lineage in data management through this informative conference talk by Mandy Chessell from Pragmatic Data Research Ltd. Delve into the importance of lineage in understanding data origins and processing for AI models, and discover how three open-source projects from LF AI and Data are collaborating to create an open ecosystem for lineage. Learn about the challenges in collecting and maintaining lineage information, and gain insights into how Egeria, OpenLineage, and Marquez work together to provide comprehensive lineage solutions. Understand the practical applications of lineage in data science tools, with a focus on Jupyter Notebooks as an example. Gain valuable knowledge on building trust in AI models through transparent data lineage and explore the benefits of open-source solutions in this critical aspect of data management.