In this course, you learn about data engineering on Google Cloud, the roles and responsibilities of data engineers, and how those map to offerings provided by Google Cloud. You also learn about ways to address data engineering challenges.
Overview
Syllabus
- Course Introduction
- Course Introduction
- Data Engineering Tasks and Components
- Module Introduction
- The Role of a Data Engineer
- Data Sources Versus Data Sinks
- Data Formats
- Storage Solution Options on Google Cloud
- Metadata Management Options on Google Cloud
- Sharing Datasets using Analytics Hub
- Lab Intro: Loading Data into BigQuery
- Loading data into BigQuery
- Quiz
- Data Replication and Migration
- Module Introduction
- Replication and Migration Architecture
- The gcloud Command Line Tool
- Moving Datasets
- Datastream
- Lab Intro: Datastream: PostgreSQL Replication to BigQuery
- Datastream: PostgreSQL Replication to BigQuery
- Quiz
- The Extract and Load Data Pipeline Pattern
- Module Introduction
- Extract and Load Architecture
- The bq Command Line Tool
- BigQuery Data Transfer Service
- BigLake
- Lab Intro: BigLake: Qwik Start
- BigLake: Qwik Start
- Quiz
- The Extract, Load, and Transform Data Pipeline Pattern
- Module Introduction
- Extract, Load, and Transform (ELT) Architecture
- SQL Scripting and Scheduling with BigQuery
- Dataform
- Lab Intro: Create and Execute a SQL Workflow in Dataform
- Create and execute a SQL workflow in Dataform
- Quiz
- The Extract, Transform, and Load Data Pipeline Pattern
- Module Introduction
- Extract, Transform, and Load (ETL) Architecture
- Google Cloud GUI Tools for ETL Data Pipelines
- Batch Data Processing Using Dataproc
- Lab Intro: Use Dataproc Serverless for Spark to Load BigQuery
- Use Dataproc Serverless for Spark to Load BigQuery
- Streaming Data Processing Options
- Bigtable and Data Pipelines
- Lab Intro: Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow
- Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow
- Quiz
- Automation Techniques
- Module Introduction
- Automation Patterns and Options for Pipelines
- Cloud Scheduler and Workflows
- Cloud Composer
- Cloud Run Functions
- Eventarc
- Lab Intro: Use Cloud Run Functions to Load BigQuery
- Use Cloud Run Functions to Load BigQuery
- Quiz
- Course Summary
- Course Summary
- Course Resources
- Your Next Steps
- Course Badge
Tags
Related Courses
-
Introduction to Data Engineering on Google Cloud - Français
-
Introduction to Data Engineering on GC - Indonesian
-
Building Batch Data Pipelines on Google Cloud
-
Introduction to Data Engineering on Google Cloud - Español
-
Google Cloud Computing Foundations: Data, ML, and AI in Google Cloud
-
Modernizing Data Lakes and Data Warehouses with Google Cloud
