ETL in Python and SQL

Introduction

• Create an ETL in Python and SQL
• Tools used in this course

1. What Are ETLs?

• What are ETLs and how do you create them?
• ETL process overview
• Exploring your data with pandas (Python) and SQL
• Understanding your data
• Challenge: Reading data using Python
• Solution: Reading data using Python

2. Extracting and Transforming Data with Python

• Loading data from different sources
• Extracting your data
• Cleaning, preprocessing data, and data formatting
• Standardization, handling duplicates, and missing values
• Challenge: Extract and transform data using pandas
• Solution: Extract and transform data using pandas

3. Loading Data into Target Systems

• Introduction to data warehouses and data lakes
• Loading data into relational databases
• Data quality checks and validation with SQL
• Challenge: Transform the data and remove duplicates and nulls
• Solution: Transform the data and remove duplicates and nulls

4. Automating ETL Jobs: Scheduling ETL Jobs with Python

• Querying your data with SQL
• Scheduling ETL jobs with Airflow: Part 1
• Scheduling ETL jobs with Airflow: Part 2
• Challenge: Load the data into a database and automate
• Solution: Load the data into a database and automate

Conclusion

• Expand your knowledge of ETLs