Modernizing Data Pipelines with Databricks - IAS Journey

Discover how Integral Ad Science (IAS) modernized their data pipelines using Databricks in this 32-minute conference talk. Learn about IAS's journey from legacy systems to a Data Mesh paradigm on a single enterprise Lakehouse, including the migration of hundreds of complex Airflow pipelines to Databricks Workflows. Explore how this transition simplified orchestration, boosted efficiency, and enabled seamless data sharing across teams using Databricks Unity Catalog for central governance. Gain insights into the company's progress towards real-time data processing and their lessons learned during the migration process. Understand how IAS managed sensitive data, built sensors and operators for pipeline dependencies, and leveraged automation and CI/CD to maximize engineer productivity. Presented by Andy Guinther and Preethi Krishnan, Staff Data Engineers at Integral Ad Science, this talk offers valuable insights for organizations looking to modernize their data infrastructure and improve cross-functional analytics and decision-making processes.