Apache Airflow: Where Data Engineers and ML Engineers Meet

Explore the intersection of data engineering and machine learning in this 47-minute conference talk from MLOps World: Machine Learning in Production. Discover how Apache Airflow, the leading workflow management and data orchestration framework, can unify workflows and integrate data engineering with Generative AI pipelines. Learn about the complexities of moving Generative AI applications to production, including automating fine-tuning, maintaining up-to-date data, managing format changes, monitoring model quality, and addressing lineage and governance requirements. Examine a production-quality Generative AI pipeline using the open-source LLM application Ask Astro as a reference architecture. Gain insights into best practices for using Airflow and key features for day 2 Generative AI operations. Understand how Airflow's extensibility allows for customization to meet specific organizational needs, emphasizing that any Python-executable task can be incorporated into Airflow workflows.