Application and Practice of Data Integration Technology at Xiaomi

Explore the application and practical scenarios of data integration technology at Xiaomi in this 27-minute conference talk by Zheng Junyuan, a senior software development engineer. Gain insights into the evolution of Xiaomi's data platform, self-developed middleware engine capabilities, and platformization practices. Learn about the evolution route, overall service architecture, and service scale of data integration technology, as well as solutions for offline/real-time synchronization scenarios, multi-source heterogeneous synchronization, and link exception fault tolerance. Discover the implementation of stream-batch integration, engine task orchestration models, and optimization rules in practice. Understand the principles, technical capabilities, and design intentions behind Xiaomi's self-developed distributed engine. Examine platformization capabilities, including data quality and link monitoring, integration of heterogeneous data platforms, link data correction, and resource estimation. Delve into the practical aspects of data integration in the engine platformization process, providing valuable insights for professionals in the big data and distributed computing fields.