Scaling Real-Time Healthcare Data Processing for Veterans Affairs

Explore a 33-minute conference talk on scaling real-time healthcare data processing for the U.S. Department of Veterans Affairs (VA). Learn about the Electronic Health Record Modernization Data Syndication initiative, which aims to migrate VA data to the cloud for improved accessibility and analysis. Discover how Azure Databricks and its Lakehouse architecture are central to the project's success, enabling robust pipelines that ingest hundreds of terabytes of historical data and employ structured streaming for real-time incremental data processing. Understand the optimization strategies, including Change Data Feed, Predictive IO, and Photon, that have reduced ETL time by over 85%, empowering the VA to deliver agile and responsive care to veterans. Gain insights from Kash Sabba, Sr. Consultant at Microsoft, and R Spencer Schaefer, Chief AI Officer VISN 15 at the U.S. Department of Veteran Affairs, as they discuss how this initiative supports over 9 million veterans across 172 medical centers and 1,200 clinics, processing 40-60 million daily patient transactions.