Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Deploy LLM to Production on Single GPU - REST API for Falcon 7B with QLoRA on Inference Endpoints

Venelin Valkov via YouTube

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Learn how to deploy a fine-tuned Falcon 7B language model with QLoRA to production using HuggingFace Inference Endpoints. Follow along as the process of merging the QLoRA adapter with the base model, pushing it to the HuggingFace Hub, and setting up a REST API is demonstrated. Gain insights into creating custom handlers for inference endpoints and testing the deployed model. This comprehensive tutorial covers everything from initial setup in Google Colab to the final steps of testing the REST API, providing a practical guide for deploying large language models in production environments.

Syllabus

- Introduction
- Text Tutorial on MLExpert.io
- Google Colab Setup
- Merge QLoRA adapter with Falcon 7B
- Push Model to HuggingFace Hub
- Inference with the Merged Model
- HuggingFace Inference Endpoints with Custom Handler
- Create Endpoint for the Deployment
- Test the Rest API
- Conclusion

Taught by

Venelin Valkov

Reviews

Start your review of Deploy LLM to Production on Single GPU - REST API for Falcon 7B with QLoRA on Inference Endpoints

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.