Explore the world of LLMOps through a 30-minute video focusing on quantization models and inference using ONNX Generative Runtime. Learn how to install ONNX runtime with GPU support and perform inference with a generative model, specifically using a Phi3-mini-4k quantized to 4int. Dive into the process of converting an original Phi3-mini-128k into a 4int quantized version using the ONNX runtime. Access the accompanying notebook on GitHub to follow along and gain hands-on experience in this cutting-edge area of data science and machine learning.
LLMOps: Quantization Models and Inference with ONNX Generative Runtime
The Machine Learning Engineer via YouTube
Overview
Syllabus
LLMOps: Quantization models & Inference ONNX Generative Runtime #datascience #machinelearning
Taught by
The Machine Learning Engineer
Related Courses
-
LLMOPs: Multimodal Prompting and Inference with Phi-3 Vision 128K Instruct on CPU - ONNX 4-Bit Quantization in C#
-
LLMOps: Comparison of OpenVino, ONNX, TensorRT, and PyTorch Inference
-
LLMOPs - Inference in CPU with Phi3 4k Instruct ONNX 4-bit Model Using C#
-
Quantization Fundamentals with Hugging Face
-
MLOps: Comparing Microsoft Phi3 Mini 128k in GGUF, MLFlow, and ONNX Formats
-
LLMOps: Using Nvidia TensorRT SDK for GPU Inference
