Building a Petabyte-Scale Cybersecurity Data Mesh in Azure with Delta Lake - HSBC Case Study
Databricks via YouTube
Overview
Syllabus
Intro
The scale of the problem
Cybersecurity Data Challenges
Don't Break DIKW
Cyber Security Science & Analytics
Azure Landing Zone Overview
Landing Zone Pattern Landing zone subscriptions
Central Delta Lake Central Delta Lake subscriptions
Data Ingestion On-premises Data External Data Ingest Transfer subscriptions subscriptions
Core Data Processing
Data Mesh Core Principles
Limitations to Data Mesh Implementation
Data Product Teams
Hybrid Distributed Data Mesh
Scalable, collaborative ETL
Terraform Managed Databricks Workspaces
End to End Data Pipeline
ETL Core Code
DLT Pipeline Flow View
Data Quality Overview
Data Quality Details
Data Quality: File Arrival Checks Workflow
Taught by
Databricks