Modern Data Orchestration: Best Practices and Real-World Use Cases

Explore advanced techniques and best practices for elevating your data pipeline game in this practical talk. Dive into real-world use cases, examining patterns for data pipelines using Airflow with Spark, DBT, and Polars. Learn strategies to avoid dependencies management in Airflow and reuse DAG templates across your organization. Delve into fundamental concepts of data pipelines, including data lineage, observability, metadata, quality, and auditing, and discover how to integrate these elements effectively. Master the art of writing clean code for data pipelines using the Factory Design Pattern with spark-submit, Airflow, and KubernatesPodOperator. Gain insights into Airflow alternatives like Dagster and Mage for your data architecture. Led by Riccardo Amadio, a Senior Data Engineer at Agile Lab, this 26-minute presentation offers a no-nonsense approach to modern data orchestration.