How We Drained Every Backbone Router Simultaneously

Explore the critical events and aftermath of Meta's severe 6-hour outage on October 4, 2021, in this SREcon22 EMEA conference talk. Delve into the chain of events that led to the simultaneous draining of every backbone router, causing a cascading disruption of network traffic and halting services. Examine how routine infrastructure maintenance on backbone routers inadvertently triggered this major incident. Gain insights into the impact on internal tools and systems used for daily operations. Learn about the company's post-incident reflections, the importance of continuous validation of support structures, disaster recovery capabilities, tooling, and processes. Discover how Meta is applying these lessons to prepare for future challenges in maintaining large-scale network infrastructure.