Building a Petabyte-Scale Cybersecurity Data Mesh in Azure with Delta Lake - HSBC Case Study

Explore a 33-minute conference talk detailing HSBC's journey in constructing a petabyte-scale cybersecurity data mesh using Azure and Delta Lake. Discover how HSBC tackled unique cybersecurity challenges, including high data volumes and strict regulatory requirements, to transform disparate data silos into a unified, scalable platform. Learn about the innovative infrastructure and architecture employed, from landing zone concepts and secure access workstations to data lake structures and isolated data ingestion. Gain insights into the hybrid data mesh leveraging Delta Lake, resulting in a flexible, secure, self-service environment that empowers cyber analysts. Delve into topics such as Azure Landing Zone overview, data ingestion processes, core data processing, and the implementation of data mesh principles. Understand the limitations faced and solutions devised, including the creation of data product teams and the use of Terraform-managed Databricks workspaces. Examine the end-to-end data pipeline, ETL core code, and data quality measures implemented to ensure robust and reliable cybersecurity analytics.