All Your Queues Are Belong to Us - The Hunt for a Network Bug in the Kernel

Embark on a thrilling journey through a complex network performance investigation in this 35-minute conference talk by Laurent Bernaille and Eric Mountain from Datadog. Discover how they uncovered lower network throughput on specific Kubernetes clusters, leading to an in-depth exploration of Linux kernel internals. Follow their step-by-step debugging process as they trace packet paths, utilize bpftrace, and ultimately identify a kernel bug affecting transmit queue usage. Learn about their collaboration with the Cilium team to develop an eBPF-based mitigation and the subsequent upstream kernel fix. Gain valuable insights into advanced troubleshooting techniques and the intricacies of network performance optimization in Kubernetes environments.