OpenLineage: An Open Standard for Data Lineage

Explore the world of data lineage in this 40-minute conference talk from ApacheCon 2022. Delve into the challenges of managing complex data pipelines and discover how OpenLineage, an open framework for collecting lineage metadata, can provide solutions. Learn about the importance of data lineage in modern data stacks, understand the basics of OpenLineage's data model and metadata collection process, and get introduced to Marquez, an OpenLineage metadata server. Gain insights on tracking job failures, ensuring data freshness and quality, predicting the impact of changes, and visualizing your entire pipeline through lineage graphs. Perfect for data professionals looking to enhance their understanding of data relationships and improve pipeline management.