Explore the process of converting a LLAMA3.1 8B model to OpenVino IR format and quantizing it to 4-bit integer precision in this 19-minute video tutorial. Learn how to perform inference on CPU using the quantized model. Access the accompanying notebook on GitHub to follow along with the step-by-step instructions for model conversion, quantization, and inference using the OpenVino Toolkit. Gain practical insights into LLMOps techniques for optimizing large language models for efficient deployment on CPU hardware.
LLMOps: OpenVino Toolkit - Quantizing LLama3.1 8B to 4int and CPU Inference
The Machine Learning Engineer via YouTube
Overview
Syllabus
LLMOps: OpenVino Toolkit quantizar 4int LLama3.1 8B e Inferencia CPU #datascience #machinelearning
Taught by
The Machine Learning Engineer
Related Courses
-
LLMOps: Comparison of OpenVino, ONNX, TensorRT, and PyTorch Inference
-
Converting Qwen2-VL 2B Model to OpenVino IR Format for CPU Inference
-
Converting Alibaba Cloud Qwen2-VL Model to OpenVino IR Format - Spanish Tutorial
-
LLMOps: Comparación de Openvino, ONNX, TensorRT y Pytorch para Inferencia
-
LLMOPs - Inference in CPU with Phi3 4k Instruct ONNX 4-bit Model Using C#
-
LLMOps: Quantization Models and Inference with ONNX Generative Runtime
