Learn how to create an end-to-end data engineering project using open tools from the modern data stack to turn scattered data into a model that drives insights and decision-making.
Overview
Syllabus
Introduction
- Transform complex data into insights
- What you should know
- Project architecture overview
- Project setup
- Understanding the Big Star Collectibles database
- Setting up your data warehouse
- Getting started with ELT tools: An introduction to Airbyte
- Deploying Airbyte for data synchronization
- Setting up sources and destinations in Airbyte
- Establishing connections in Airbyte
- Synchronizing and navigating through data
- Introduction to data modeling with dbt
- Understanding the structure of a dbt project
- Initiating your dbt project
- Configuring data sources in dbt
- Challenge: Add a freshness check
- Solution: Add a freshness check
- Creating and customizing your dbt models
- Reviewing and executing dbt
- Securing your data with dbt tests
- Challenge: Add tests to the Marts model
- Solution: Add tests to the Marts model
- Automating documentation in dbt
- Completing your dbt project: A full development cycle
- Introduction to data orchestration with Dagster
- Integrating dbt models with Dagster assets
- Integrating Airbyte connections with Dagster assets
- Materializing assets using Dagit
- Challenge: Add a schedule to your data pipeline
- Solution: Add a schedule to your data pipeline
- An evolving field
Taught by
Thalia Barrera
