Learn how to leverage Envoy Proxy's unique architecture and capabilities to build a robust LLM gateway in this 39-minute conference talk. Discover why Envoy Proxy surpasses traditional proxies like NGINX through its extensibility features, including filter architecture, callout architecture (ext-proc, ext-auth), and dynamic library loading capabilities. Explore practical applications of Envoy proxy for essential LLM management functions including credential management, prompt guarding/decorating, content safety analysis, usage controls, context-aware failover, and observability. Gain insights into implementing secure, cost-effective, and content-aware control over LLM usage, leveraging Envoy's high-performance async core written in C++ for deployment as an ingress, egress, and mesh gateway. Particularly valuable for developers, architects, and technology professionals seeking solutions for LLM implementation challenges and infrastructure platform optimization.
Harnessing the Power of Envoy Proxy for Building an LLM Gateway
CNCF [Cloud Native Computing Foundation] via YouTube
Overview
Syllabus
Harnessing the Power of Envoy Proxy for Building an LLM Gateway - Idit Levine, Solo.io
Taught by
CNCF [Cloud Native Computing Foundation]