Overview

This course will guide you on how to prepare for the DP-203: Data Engineering on Microsoft Azure certificate exam. By the end of this course, you will be able to: - Explain how to design data for analysis using Azure Databricks, Apache Spark, and Azure Synapse pipelines. - Describe how to ingest, clean, and transform data using Azure Synapse. - Identify data processing solutions using Azure Databricks and manage pipelines in Azure Synapse pipelines. - Discuss the steps to secure, optimize, and monitor data storage using Azure Synapse Analytics. This course is designed for IT professionals who want to prepare for the Microsoft DP-203 exam and demonstrate their expertise in creating analytical solutions by integrating, transforming, and consolidating data from multiple data sources, such as structured, unstructured, and streaming data systems. According to Microsoft, candidates for the DP-203 exam should have experience with operationalization of data pipelines and ensure that data stores are high-performing, efficient, organized, and reliable, given a set of business requirements and constraints. You should be able to identify and troubleshoot operational and data quality issues, and design, implement, monitor, and optimize data platforms to meet the data pipelines.

Syllabus

Designing and implementing data storage and data exploration layer

In this module, you will be introduced to the Azure tools available for designing and implementing data storage and data exploration layers. You will gain insight into the three important data engineering tools: Azure Synapse Analytics, Azure Databricks, and Azure Data Lake. You will gain insight into the key concepts and strategies of data partitioning for files, focusing on both Azure Synapse Analytics and Azure Data Lake Storage Gen2. Additionally, you will learn to browse and search metadata in Microsoft Purview Data Catalog for data exploration. The module also delves into the process of creating and executing a compute solution in Azure, which will enable you to make informed decisions regarding data partitioning, enhancing data cataloging capabilities, and efficiently managing data storage and processing resources in Azure.

Developing data processing

In this module, you will gain an understanding of data ingestion and transformation techniques. You will also learn about Transact SQL for data transformation, Azure Synapse Pipelines for ETL, Apache Spark for processing, and exploratory data analytics. The module also discusses data loading with PolyBase, the creation of data pipelines, and integration with Jupyter and Python notebooks. Additionally, it delves into configuring data snapshots with Delta Lake and building stream processing solutions using Spark Structured Streaming. Finally, you will gain a comprehensive understanding of data processing options and techniques.

Securing, monitoring, and optimizing data storage and data processing

In this module, you will learn about the importance of implementing robust data security measures in Azure, encompassing data encryption, access control through Azure RBAC, and secure data handling. Additionally, you can learn the steps to monitor data storage, optimize query performance through indexers and caching, and troubleshoot Spark job and pipeline run failures effectively. This knowledge is vital for maintaining efficient data storage and retrieval, contributing to improved data processing and analytics. You will also gain insight into two critical areas of Azure data management: Data security and performance optimization. You will also learn some insights on implementing robust data security measures to protect sensitive information effectively.