Get an introduction to Apache Airflow—its uses, structure, how to get it up and running, and how to create and execute workflows.
Overview
Syllabus
Introduction
- An overview of Apache Airflow
- Apache Airflow concepts and architecture
- Initial setup: macOS and Windows
- Understanding Airflow configuration
- Running Airflow
- Navigating and exploring the Airflow UI
- Creating your first DAG
- Running your first DAG
- Creating a DAG using the with keyword
- Specifying task dependencies
- Using the Bitshift operators for dependency specification
- Using bash scripts in the DAG
- Introducing the Python operator
- Python operators and dependencies
- Passing parameters to Python callables
- Introducing XCom
- Using XCom to pass values
- Implementing a data transformation DAG pipeline, part 1
- Implementing a data transformation DAG pipeline, part 2
- Setting up the SQLite DB and connection
- Introducing the SQLite operator
- Implementing a SQL-based DAG pipeline, part 1
- Implementing a SQL-based DAG pipeline, part 2
- Introducing branching
- Conditional branching with variables
- Task groups and edge labels
- Catch up with all previously scheduled runs
- Backfill scheduled runs
- Summary and next steps
Taught by
Janani Ravi