Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Building a Petabyte-Scale Cybersecurity Data Mesh in Azure with Delta Lake - HSBC Case Study

Databricks via YouTube

Overview

Explore a 33-minute conference talk detailing HSBC's journey in constructing a petabyte-scale cybersecurity data mesh using Azure and Delta Lake. Discover how HSBC tackled unique cybersecurity challenges, including high data volumes and strict regulatory requirements, to transform disparate data silos into a unified, scalable platform. Learn about the innovative infrastructure and architecture employed, from landing zone concepts and secure access workstations to data lake structures and isolated data ingestion. Gain insights into the hybrid data mesh leveraging Delta Lake, resulting in a flexible, secure, self-service environment that empowers cyber analysts. Delve into topics such as Azure Landing Zone overview, data ingestion processes, core data processing, and the implementation of data mesh principles. Understand the limitations faced and solutions devised, including the creation of data product teams and the use of Terraform-managed Databricks workspaces. Examine the end-to-end data pipeline, ETL core code, and data quality measures implemented to ensure robust and reliable cybersecurity analytics.

Syllabus

Intro
The scale of the problem
Cybersecurity Data Challenges
Don't Break DIKW
Cyber Security Science & Analytics
Azure Landing Zone Overview
Landing Zone Pattern Landing zone subscriptions
Central Delta Lake Central Delta Lake subscriptions
Data Ingestion On-premises Data External Data Ingest Transfer subscriptions subscriptions
Core Data Processing
Data Mesh Core Principles
Limitations to Data Mesh Implementation
Data Product Teams
Hybrid Distributed Data Mesh
Scalable, collaborative ETL
Terraform Managed Databricks Workspaces
End to End Data Pipeline
ETL Core Code
DLT Pipeline Flow View
Data Quality Overview
Data Quality Details
Data Quality: File Arrival Checks Workflow

Taught by

Databricks

Reviews

Start your review of Building a Petabyte-Scale Cybersecurity Data Mesh in Azure with Delta Lake - HSBC Case Study

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.