Explore the innovative LoRAX (LoRA eXchange) LLM inference system in this informative conference talk. Learn how LoRAX enables users to pack thousands of fine-tuned "LoRA" adapters into a single GPU, significantly reducing serving costs compared to dedicated deployments per fine-tuned model. Discover the key features of this open-source, commercially free, and production-ready system, including pre-built docker images and Helm charts. Delve into the core concepts that make LoRAX the most cost-effective and efficient solution for serving fine-tuned LLMs in production, such as Dynamic Adapter Loading, Heterogeneous Continuous Batching, and Adapter Exchange Scheduling. Gain insights into how these techniques optimize latency, throughput, and resource utilization while managing multiple concurrent adapters on a single GPU.
LoRAX - Serving Thousands of Fine-Tuned LLMs on a Single GPU
Overview
Syllabus
LoRAX: Serve 1000s of Fine-Tuned LLMs on a Single GPU - Travis Addair, Predibase, Inc.
Taught by
Linux Foundation
Tags
Related Courses
-
Efficiently Serving LLMs
-
dLoRA - Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving
-
The Future is Fine-Tuned - Deploying Task-Specific LLMs
-
Delivering Domain-Specific LLMs with GPU Serving - Case of IFC MALENA
-
Fine-Tuning Llama 3 on a Custom Dataset for RAG Q&A - Training LLM on a Single GPU
-
Fine-Tuning LLM with QLoRA on Single GPU - Training Falcon-7b on ChatBot Support FAQ Dataset
