Data Alchemy: Transforming Raw Data to Gold with Apache Hudi and DBT

Watch a 29-minute conference talk from OSA Con 2023 where Nadine Farah explores how to build an efficient medallion architecture using Apache Hudi and DBT. Learn about transforming raw operational data into refined analytics-ready tables through a series of processing stages. Discover how Apache Hudi's transactional data lake platform enables streaming upserts and incremental processing, while DBT's incremental loading feature optimizes resource usage by processing only new or updated records. Understand the challenges of building low-latency medallion architectures, explore Hudi's Change Data Capture (CDC) feature for processing change records, and learn practical techniques for leveraging DBT to transform data modifications across bronze to gold tables. Gain insights into implementing these technologies to create efficient, scalable data processing pipelines used by some of the largest transactional data lakes in the industry.