Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Observability for Data Pipelines with OpenLineage

Databricks via YouTube

Overview

Explore the critical importance of observability in data pipelines through a 24-minute video presentation from Databricks. Dive into the world of OpenLineage, an API standardizing metadata collection across ecosystems, and discover how it enhances data operations, governance, and security. Learn about Marquez, an open-source project that implements OpenLineage API to provide visibility into dependencies across organizations and technologies. Gain insights into how collecting lineage metadata during pipeline execution can improve auditability, reliability, and timeliness of data operations in fast-paced environments. Understand the role of metadata in creating reliable data products and how OpenLineage reduces complexity in lineage information collection. Examine the Spark integration and its purpose in enhancing data pipeline observability.

Syllabus

Intro
Metadata
OpenLineage
Purpose
Spark Integration

Taught by

Databricks

Reviews

Start your review of Observability for Data Pipelines with OpenLineage

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.