Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Linux Foundation

Effortless Scalability: Orchestrating Large Language Model Inference with Kubernetes

Linux Foundation via YouTube

Overview

Explore the intricacies of deploying and orchestrating large open-source inference models on Kubernetes in this 27-minute talk from the Linux Foundation. Dive into automating the deployment of heavyweight models like Falcon and Llama 2 using Kubernetes Custom Resource Definitions (CRDs) for seamless management of large model files through container images. Learn about streamlining deployment with an HTTP server for inference calls, eliminating manual tuning of deployment parameters with preset configurations, and auto-provisioning GPU nodes based on specific model requirements. Discover how to empower users to deploy containerized models effortlessly by providing pod templates in the workspace custom resource inference field, allowing the controller to dynamically create deployment workloads utilizing all GPU nodes. Gain insights into optimizing resource utilization and simplifying the deployment process for large language model inference in the rapidly evolving AI/ML landscape.

Syllabus

Effortless Scalability: Orchestrating Large Language Model Inference w... Joinal Ahmed & Nirav Kumar

Taught by

Linux Foundation

Reviews

Start your review of Effortless Scalability: Orchestrating Large Language Model Inference with Kubernetes

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.