Building Data Lakehouse from Scratch - End-to-End Data Engineering Project

Overview

Learn to design, implement, and maintain a secure, scalable, and cost-effective data lakehouse architecture in this comprehensive end-to-end data engineering project. Explore advanced techniques using Apache Spark, Apache Kafka, Apache Flink, Delta Lake, AWS, and open-source tools to unlock data's full potential through analytics and machine learning. Dive into modern system architectures, create databases, utilize Glue crawlers, and automate data orchestration with Lambda functions on AWS Cloud. Gain hands-on experience in coding, optimizing, and verifying results while mastering the intricacies of building a robust data lakehouse from scratch.

Syllabus

Introduction
The system architecture
The modern system architecture
Implementation of the Current Data Lakehouse on AWS Cloud
Creating Databases for Data Lakehouse
Using Glue crawler for Data Lakehouse
Using Lambda function to automate data orchestration on AWS Cloud
Coding the Lambda function
Optimising Lambda Function
Verification of Results
Outro