Explore the intricacies of building a ChatGPT-style LLM AI infrastructure using Kubernetes in this comprehensive video featuring John McBride. Delve into the challenges and solutions of deploying open-source AI technologies at scale, with a focus on Kubernetes as a platform for running compute-intensive tasks. Learn about the decision-making process behind choosing TimeScaleDB for storing time-series data and vectors, and gain insights into migrating from OpenAI to an open-source large language model inference engine. Discover the importance of selecting the right level of abstraction, understanding trade-offs, and evaluating language model performance. The video also covers practical aspects such as deploying Kubernetes, setting up node groups with GPUs, and using VLLM as the inference engine. Whether you're a startup considering Kubernetes adoption or an experienced developer looking to optimize AI infrastructure, this talk provides valuable takeaways on building and managing AI-enabled applications at scale.
Overview
Syllabus
John McBride
Introduction and Background
Summary of the Blog Post
The Role of Kubernetes in AI-Enabled Applications
The Use of TimeScaleDB for Storing Time-Series Data and Vectors
Migrating to an Open-Source LLM Inference Engine
Deploying Kubernetes and Setting Up Node Groups
Choosing VLLM as the Inference Engine
The Migration Process: Deploying Kubernetes and Setting Up Node Groups
Choosing the Right Level of Abstraction
Challenges in Evaluating Language Model Performance
Considerations for Adopting Kubernetes in Startups
Taught by
Tejas Kumar
Related Courses
-
Load-Aware GPU Fractioning for LLM Inference on Kubernetes
-
Optimizing LLM Inference with AWS Trainium, Ray, vLLM, and Anyscale
-
Best Practices for Deploying LLM Inference, RAG and Fine-Tuning Pipelines on Kubernetes
-
Effortless Scalability - Orchestrating Large Language Model Inference with Kubernetes
-
Building a Multi-Cluster Privately Hosted LLM Serving Platform on Kubernetes
-
Effortless Scalability - Orchestrating Large Language Model Inference with Kubernetes
