Bloomberg's Journey to a Multi-Cluster Workflow Orchestration Platform
CNCF [Cloud Native Computing Foundation] via YouTube
Overview
Explore Bloomberg's journey in developing a multi-cluster workflow orchestration platform for automation, data pipelines, model training, and general utility compute. Learn about the challenges faced with rapid adoption, including increased latency and query timeouts. Discover the evolution from an initial implementation using bespoke in-cluster pull agents and a source-of-truth to a current approach leveraging a highly-available PostgreSQL database and Kine. Gain insights into the investigation of related projects and inspirations drawn from Karmada, OCM, and others. Understand the lightweight Kubernetes API server interface design that enables efficient multi-cluster federation for deploying heterogeneous resources.
Syllabus
Bloomberg's Journey to a Multi-Cluster Workflow Orchestration Platform - Yao Lin & Reinhard Tartler
Taught by
CNCF [Cloud Native Computing Foundation]