ModelMesh: Scalable AI Model Serving on Kubernetes

Explore the scalable deployment of AI models on Kubernetes using ModelMesh, the multi-model serving backend for KServe, in this conference talk. Learn how to overcome resource limitations and efficiently manage numerous models at scale. Discover ModelMesh's distributed LRU cache for intelligent model loading and unloading, as well as its routing capabilities for balancing inference requests. Gain insights into the latest major release (v0.10) and its integration with KServe. Understand the advantages of ModelMesh's small control-plane footprint and its ability to host multiple models while maximizing cluster resources and minimizing costs. Explore newly-supported model runtimes like TorchServe and the capability for runtime-sharing across namespaces. Dive into the ModelMesh architecture, monitoring techniques using Prometheus, and practical examples of custom runtimes and inference services through a live demonstration.