Learn how to convert the LLAMA3.2 3 Billion parameter model to OpenVino IR format and quantize it to 4-bit integer precision. Follow along as the process of model conversion and quantization is demonstrated step-by-step. Discover how to perform inference on a CPU using Chain of Thought (CoT) prompts with the optimized model. Access the accompanying Jupyter notebook for hands-on practice and deeper understanding of the LLMOps techniques covered in this 26-minute tutorial on data science and machine learning.
LLMOps: OpenVino Toolkit Quantization 4int LLama 3.2 3B and Inference on CPU
The Machine Learning Engineer via YouTube
Overview
Syllabus
LLMOps: OpenVino Toolkit quantization 4int LLama3.2 3B, Inference CPU #datascience #machinelearning
Taught by
The Machine Learning Engineer
Related Courses
-
LLMOps: Comparison of OpenVino, ONNX, TensorRT, and PyTorch Inference
-
LLMOps: Comparación de Openvino, ONNX, TensorRT y Pytorch para Inferencia
-
MLOps: OpenVino Quantized Pipeline for Grammatical Error Correction
-
LLMOps: Converting Video Classifier (ViViT) to ONNX and Inference on CPU
-
LLMOps: Quantization Models and Inference with ONNX Generative Runtime
-
LLMOPs: Multimodal Prompting and Inference with Phi-3 Vision 128K Instruct on CPU - ONNX 4-Bit Quantization in C#
