Learn to harness the power of Apache Airflow with Python in this comprehensive tutorial. Explore essential concepts including email operators, Docker Compose integration, DAG organization, REST API triggering, user authentication, and achieving parallelism in ETL workflows. Dive into advanced topics such as Task Groups, scheduler reliability, AWS S3 log storage, and dynamic DAG addition using AWS Lambda. Master Airflow's capabilities to streamline and optimize your data pipeline management.
Overview
Syllabus
Learning Apache Airflow with Python in easy way in 40 Minutes.
How to use Apache Airflow Email Operator in docker compose file Python.
Apache Airflow + Flower + Celery Worker with Docker Compose.
organizing Dags and Helper function in Airflow Python.
Trigger Airflow Dag Via REST API tutorials.
How to correctly do Airflow User Authentication.
Learn how you can achieve parallelism when creating ETL or workflows using Pandas & Airflow.
Airflow2 docker compose with celery Learn about Task Group in Airflow.
Airflow 2: What happens if the scheduler goes down in Airflow? | Docker Compose.
Apache Airflow2: Storing logs in AWS S3 | docker compose |.
Airflow 2 Adding a new Dag Dynamically to Dags Folder with AWS S3 and Lambda.
Taught by
soumilshah1995
