Designing Data Warehouse from Scratch - End-to-End Data Engineering

Overview

Learn how to design and implement a data warehouse from scratch in this comprehensive end-to-end data engineering tutorial. Explore the process of creating a logical data warehouse architecture, setting up an AWS Redshift cluster, converting logical models to physical implementations, loading data, and implementing the medallion architecture. Gain hands-on experience with AWS services, including VPC, Redshift, Glue Data Catalog, and S3, while also learning to use tools like DBeaver and dbt for data modeling and transformation. Follow along as the instructor guides you through each step, from designing dimension and fact tables to creating silver and gold models in a real-world business scenario.

Syllabus

Introduction
System Prerequisites
Steps Involved in Designing a Data Warehouse
The Business Usecase
Designing the Logical Architecture
Creating a VPC on AWS
Creating Redshift Data Warehouse Cluster
Creating Subnet Group on AWS
Creating Security Group and allowing external connections on AWS
Connecting to Redshift Cluster with DBeaver
Connecting to Redshift Cluster with Redshift Query Editor
Creating Dimensions and Fact data
Loading data into Data Warehouse
Creating AWS Data Catalog DB and Tables
Connecting to Redshift to AWS Glue Data Catalog
Creating DBT project
Configuration connections to Redshift from DBT
DBT Project configuration with Variables and Schema
Creating Silver Dimension models
Creating Silver Fact models
Creating Gold Dimension and Fact Models
Other course information