Elephant on Wheels - Petabyte-scale AI @ LinkedIn

Explore how LinkedIn leverages Kubernetes for petabyte-scale AI workloads in this conference talk. Discover the journey from proof of concept for Jupyter notebooks to a key infrastructure for model training and serving. Learn about Kube2Hadoop, a scalable and secure open-source integration that allows AI and non-AI workloads to access HDFS securely. Gain insights into how LinkedIn's AI modelers use data securely in model exploration and training with KubeFlow components. Understand the prototyping of a multilevel scheduler on top of Kubernetes and YARN clusters in the cloud, enabling intelligent job routing and facilitating workflows across different cluster types. Dive into topics such as Hadoop vs Kubernetes, hadoop delegation token, cube2hadoop components and workflow, authentication mechanisms, and future improvements. Witness a demo showcasing the practical application of these concepts in LinkedIn's AI infrastructure.