Explore a comprehensive framework for data lake ingestion using Azure Data Factory in this 52-minute conference talk from PASS Data Community Summit. Dive into data strategy and its impact on framework design, covering the development of a metadata database using Data Vault modeling, metadata collection techniques, and the creation of a fully automated Data Factory pipeline. Learn about data orchestration, ETL/ELT automation, data lake organization, and dynamic SQL implementation in ADF. Gain insights into ingestion conceptual architecture, schema loading, and other key features to enhance your data engineering skills and optimize your Azure-based data solutions.
Overview
Syllabus
Intro
Upcoming Sessions
Call for Presenters
PASS Summit 2019 Introducing Learning Pathways
PASS Summit 2019 Future-Proof Your Data Career
Virtual Group Giveaways
Problem Statement?
AGENDA DATA FACTORY INGESTION FRAMEWORK 1 Intro to Data Factory
What is Data Factory? ? Data Orchestration - Originally developed for data integration across the cloud where storage and compute are separated
Azure Data Factory
Data Factory Sources
ETL Automation - Traditional
ELT Automation - Modern Extract
Data Ingestion Components Data Lake Ingestion takes into account the following components...
Data Lake Organization RAW ZONE Delta Captures /ServerName
Ingestion Conceptual Architecture
Data Vault Example
Learn Data Vault
Loading the Model
ADF: Schema Loader
ADF: Data Loader
Dynamic Linked Services FOREACH ACTIVITY
Dynamic SQL in ADF
Metadata Model Views
Other Features
CONTACT
Taught by
PASS Data Community Summit
Related Courses
-
Azure Data Factory for Beginners - Build Data Ingestion
-
Data integration at scale with Azure Data Factory or Azure Synapse Pipeline
-
Azure Data Factory For Data Engineers - Project on Covid19
-
Implement a data lakehouse analytics solution with Azure Databricks
-
Ingesting and Managing Petabytes of Data with Azure Data Factory
-
Moving Data with Azure Data Factory
