Code Once, Use Often - Declarative Data Pipelines

Explore declarative data pipelines in this 28-minute conference talk featuring Anthony Awuley and Carter Kilgour from Flashfood. Learn about food waste reduction efforts, Flashfood's data challenges, and the evolution of their data pipeline solutions. Discover the benefits of declarative approaches, including code reusability and automation. Gain insights into tools like airflow-declarative, SyncTable, and custom operators for extract and transform processes. Understand key lessons learned, future challenges, and the potential of Spark YAML. Conclude with a discussion on Keillor's Principles and the importance of feedback in data engineering.