Python for Data Engineering: from Beginner to Advanced

Introduction

• Welcome to the course
• What you should know
• CoderPad tour

1. Python Basics

• Introduction to Python and data engineering
• Setting up your Python environment
• Explore a Google Colab worksheet
• Variables and data types
• Operators and expressions
• Control structures
• Functions
• Modules and packages
• String manipulation
• Error handling
• Solution: String Manipulation

2. Python Collections

• Collection overview
• Python collections: Tuples
• Python collections: Lists
• Python collections: Sets
• Python collections: Dictionaries
• Solution: Analyze list

3. Python File Handling

• File I/O overview
• Working with CSV files
• Working with JSON files
• Solution: Read/Write text to file

4. pandas DataFrame API

• Introduction to pandas
• Read files as DataFrames
• Data cleaning and preprocessing
• Data manipulation and aggregation
• Data visualization
• Write DataFrames as files
• Solution: Play with pandas

5. NumPy

• Introduction to NumPy
• Array creation and attributes
• Array operations
• Indexing and slicing
• Linear algebra and statistics
• Write DataFrames as files
• Solution: NumPy Array Operation

6. OOP with Python

• Understanding classes and objects
• Implementation: Classes and objects in Python
• Understand OOP features: Abstraction, inheritance, and more
• Solution: Accessing Object attributes

7. Advanced Data Engineering

• Tips to write efficient Python code
• What is ETL in the data engineering world?
• What is Hadoop?
• Understand PySpark for data engineering
• Importance of visualization tools in DE
• On-prem vs. cloud data engineering

8. Capstone Project

• Capstone project: Retail sales analysis
• Solution: Capstone project

Conclusion

• Next steps