Hydra - A Federated Resource Manager for Data-Center Scale Analytics

Explore Microsoft's Hydra, a federated resource manager designed for data-center scale analytics, presented by Carlo Curino at NSDI '19. Dive into the challenges of scheduling exabytes of data over millions of cores daily for thousands of tenants. Learn how Hydra's federated architecture and agile control plane enable efficient task placement and rapid policy updates across tens of thousands of nodes. Discover how this system, built on Apache Hadoop YARN, has become Microsoft's primary big-data resource manager, scheduling nearly one trillion tasks and manipulating close to a Zettabyte of production data. Gain insights into the scale/utilization challenges, architecture, scheduling desiderata, policies, and solutions for handling large-scale analytics workloads in modern data centers.