Debugging Complex Kubernetes Incidents - When It's Not DNS

Dive into a complex Kubernetes incident investigation in this conference talk. Follow the journey of troubleshooting mysterious service errors during rolling updates, initially suspected to be DNS-related. Explore the debugging steps, from analyzing application behavior and DNS setup to investigating networking issues and VPC flow logs. Uncover the intricacies of ingress and egress flows, routing on nodes, and the impact of reverse path filtering. Examine the RPC setup, DNS propagation time during rollouts, and reconnection differences. Learn valuable lessons from this in-depth exploration of a challenging issue that ultimately led to a simple three-line code removal solution.