Real-time Schema Discovery in Distributed Systems

Explore real-time schema discovery in this 42-minute conference talk by Dan Selans, co-founder and CTO of Streamdal.com. Dive deep into the development of an automated schema evolution process for complex distributed systems processing over 100,000 messages per second. Learn about schema versioning, conflict detection, compatibility, and normalization without batching processes. Discover techniques for detecting schema drift and determining compatibility in high-throughput environments. Gain insights from Selans' extensive experience in software development at companies like InVisionApp, New Relic, and DigitalOcean, as well as his background in data center integration and R&D. This Data Council presentation offers valuable knowledge for data professionals seeking to enhance their understanding of real-time schema management in large-scale distributed systems.