Explore the process of converting a LLAMA3.1 8B model to OpenVino IR format and quantizing it to 4-bit integer precision in this 19-minute video tutorial. Learn how to perform inference on CPU using the quantized model. Access the accompanying notebook on GitHub to follow along with the step-by-step instructions for model conversion, quantization, and inference using the OpenVino Toolkit. Gain practical insights into LLMOps techniques for optimizing large language models for efficient deployment on CPU hardware.
LLMOps: OpenVino Toolkit - Quantizing LLama3.1 8B to 4int and CPU Inference
The Machine Learning Engineer via YouTube
Overview
Syllabus
LLMOps: OpenVino Toolkit quantizar 4int LLama3.1 8B e Inferencia CPU #datascience #machinelearning
Taught by
The Machine Learning Engineer
Related Courses
-
LLMOps: Comparison of OpenVino, ONNX, TensorRT, and PyTorch Inference
-
LLMOps: Comparación de Openvino, ONNX, TensorRT y Pytorch para Inferencia
-
LLMOPs - Inference in CPU with Phi3 4k Instruct ONNX 4-bit Model Using C#
-
LLMOps: Quantization Models and Inference with ONNX Generative Runtime
-
LLMOPs: Multimodal Prompting and Inference with Phi-3 Vision 128K Instruct on CPU - ONNX 4-Bit Quantization in C#
-
LLMOps: Converting Video Classifier (ViViT) to ONNX and Inference on CPU
