Apache Airflow Essential Training

Introduction

• Apache Airflow essential training

1. Working with the TaskFlow API

• Prerequisites
• Quick Airflow setup overview
• DAG using PythonOperators
• DAG using TaskFlow
• Passing data using XCom with operators
• Passing data using the TaskFlow API
• Tasks with multiple outputs
• Passing multiple outputs in TaskFlow

2. Performing Operations Using TaskFlow

• Branching using operators
• Branching using the TaskFlow API
• Complete branching pipeline using TaskFlow
• Interoperating and passing data between operators and TaskFlow
• Performing SQL operations with TaskFlow
• Data transformation and storage using TaskFlow

3. Integrating PostgreSQL in Airflow

• Installing PostgreSQL on macOS
• Installing PostgreSQL on WSL
• Connecting to PostgreSQL
• Using the PostgreSQL operator
• Performing PostgreSQL insert operations
• Performing PostgreSQL join operations
• A complete end-to-end pipeline with PostgreSQL
• Configuring PostgreSQL as a metadata database and using the LocalExecutor

4. Using Sensors in Airflow

• Introducing the FileSensor
• Creating a pipeline using the FileSensor
• Running DAGs with a FileSensor
• Creating and running a pipeline with SQLSensors

5. Working with Hooks in Airflow

• Setting up for a PostgreSQL pipeline with hooks
• Creating and running a pipeline with PostgresSQL hooks
• Setting up access to Amazon S3 buckets
• Setting up a connection to Amazon S3 buckets
• Creating and running a pipeline with an S3 hook

Conclusion

• Summary and next steps